The former leader of Pembrokeshire County Council has said Fishguard is losing out to Pembroke Dock because the A40 road to the port is so bad.
Councillor John Davies wants the Welsh Government to fund the widening of the A40 all the way to Fishguard.
The Stena ferry company has dropped a £5m upgrade for Fishguard, described as a "worrying" move by local politicians.
But Welsh Government minister Baroness Eluned Morgan said there was less money for roads because of austerity.
Mr Davies called the A40 to Fishguard a "country lane" and said politicians should "hang their heads in shame".
"There is a chronic lack of investment in the A40. If you are a haulier, you decide in St Clears whether to go left or right," he said.
"If you go left, you go down the A477 and you're in Pembroke Dock in no time at all. Or do you go down the windy elongated route down the A40?"
Stena Line had planned to upgrade Fishguard, replacing the flexible ship-to-shore bridge that allows vehicles on and off its ferries.
It scrapped the plan in June although it said it was still "fully committed" to using the port.
Mr Davies said the Welsh Government should fund turning the A40 into a dual carriageway to Haverfordwest and eventually through to Fishguard as well.
In 2015 the Welsh Government commissioned the A40 St Clears to Haverfordwest study.
The report said dualling would provide significant benefits but it did not find the investment justified with a journey-time saving of 11 minutes at a cost of more than £400m.
Labour's Baroness Morgan acknowledged that a dual carriageway A40 would help Fishguard port, but said the impact of austerity meant there was less money available for road construction.