Fishermen have warned there will be no industry left in Wales within three years unless the Welsh Government changes its management plan.
Tight restrictions on when and where shell fishing is allowed within some conservation areas has resulted in a decline in the number of working boats.
Fishermen want new areas opened up along the Welsh coast.
The Welsh Government said it was committed to a "strong and thriving" fishing industry after Brexit.
Scallop fisherman Richard Kynoch, 54, from Fishguard, has started spending winters working in the Isle of Man, the English channel and France to escape the restricted areas.
He said: It was hard working away from home.
"It had a big impact on my family, my marriage broke up and my children have suffered because of it."
The West Wales Shell Fisherman's Association said the number of scallop vessels had dropped by a third in the past five years, with only 10 Welsh boats now operating.
"We're forced to go further out to find viable scallop fishing grounds," a spokesman said.
"We're living like tramps on our boats. For our own safety we shouldn't be doing what we're doing, but we're forced to."
Cardigan Bay is the most productive scallop ground within Welsh waters, but it is also a significant habitat for the bottlenose dolphin.
In 2008, fishing in the area was halted after an area stretching from Ceibwr Bay in Pembrokeshire to Aberarth in Ceredigion, and extending 12 miles (19km) out to sea, was designated an Special Area of Conservation (SAC).
Dolphin watching and wildlife tourism has been a major draw ever since and a study by Aberystwyth University in 2013 showed it generated £4.9 million.
But there were fears the scallop industry, which dredges along the sea bed, could disturb the dolphin's habitat and fishermen had to move to a small area three miles (5km) off shore.
The West Wales Shell Fisherman's Association claims the ground is no longer viable, and it is calling on the Welsh Government to open up a different area within the SAC for them to fish.
Prof Mike Kaiser from Bangor University, who has researched the marine habitat within Cardigan Bay, said a study he did in 2015 found there were areas that could be opened up on a rotation basis without having a harmful effect on the environment.
He said: "We looked at fishing different intensities and found it certainly can be managed in a very sustainable way.
"The seabed recovers quickly. The science is good. We just need the government to put in place a fisheries management plan."
The Welsh Government said it had listened to the West Wales Shell Fisherman's Association's concerns and said it was committed to ensuring Wales has a strong and thriving fishing industry post Brexit.