Drivers will face long delays at ports and tailbacks on the roads if Brexit is not handled properly, an assembly committee has warned.
Its report said urgent action was needed to avoid disruption.
AMs on the cross-party external affairs committee also said they were disappointed by how little contact there had been between the Welsh and Irish governments to discuss Brexit.
The Welsh Government said it would look at the report and respond formally.
The committee's report has looked at the impact of leaving the EU on Wales' 32 ports - seven of which are considered major ports.
It said having a soft border in Ireland but a hard sea border between the Republic and the UK could "severely disadvantage Welsh ports and result in a loss of competitiveness leading to a displacement of traffic from Welsh ports, principally Holyhead, to ports in England and Scotland, via Northern Ireland".
The committee said it was "extremely concerned" that Welsh Economy Secretary Ken Skates has not met counterparts in the Irish government to discuss Brexit.
AMs said freight companies could send cargo to Northern Ireland from Scottish and English ports, instead of sailing directly to the Republic of Ireland from Holyhead, Pembroke and Fishguard.
It is not currently known what type of border would exist between Britain and Ireland.
The ports are estimated to contribute £1bn to Wales' economy.
"Urgent action will be necessary if our worst fears - of lengthy delays at our ports, disruption to wider supply chains and tailbacks on our roads - are to be avoided," the committee said.
There are also warnings about the "technological and logistical challenges" if new customs checks are needed and about whether ports would be able to cope with border controls.
Committee chairman David Rees said: "Welsh ports directly support 18,400 jobs and many more besides.
"At the moment, Holyhead and Fishguard both work on the premise of seamless travel from one side of the Irish Sea to the other for goods and people.
"We learned that many Welsh ports lack the physical capacity to accommodate new customs and border checks, which could have an unwelcome effect including increased delays and congestion."
Welsh Conservative leader Andrew RT Davies said the report's finding that Economy Secretary Ken Skates had not met Irish counterparts to discuss a way forward was "damning".
"It's yet more evidence of a Labour government which is happy to sit on the side-lines complaining - rather than taking a positive and proactive approach to making Brexit work for everyone," he said.
A Welsh Government spokesman said it had "set out clearly our priorities for Brexit" and stressed any changes to migration and customs rules "would have an immediate and major impact at Welsh ports".
"As the economy secretary made clear in his evidence to committee, tariffs, the wider fiscal regime and customs arrangements are reserved matters for HMRC," he said.
"We have been calling on the UK government to ensure that - if they continue to insist on an approach to Brexit which would take the UK out of both the single market and the customs union - customs arrangements, which are the ports sector's key concern, are proportionate to support so-called 'frictionless' trade and do not in any way disadvantage Welsh ports.
"We have also pressed the point that the UK government needs to provide funding for any additional requirements that Welsh ports might require as a result of arrangements post Brexit, including infrastructure."
The spokesman added it would continue to "engage closely with the sector on the potential threats and opportunities that Brexit presents for ports" and continue to work with other partners, including its counterparts in Ireland.
The UK government's Brexit department declined to comment.