Proposals for a major overhaul of how taxis work in Wales fail to tackle a "crisis" in the industry, it is claimed.
The Welsh Government wants to strip powers from councils and set-up an all-Wales licensing body.
A union has called for councils to be given powers to stop out-of-town drivers, which it said makes it harder for others to earn a living.
The Welsh Government said the concerns will be considered.
It said current laws do not reflect the fact taxis and private-hire drivers are conducting pre-booked work outside of the place they got their licence, often through smart-phone based apps.
Not everyone in the industry feels the same and some drivers argue allowing taxis to operate across council borders boosts customer choice and is essential away from Cardiff.
The system, only recently devolved and parts of which date back to the 19th Century, had envisaged that enforcing standards would take place within council boundaries.
But drivers can shop around Welsh local authorities, which may have different standards and charge different fees.
A white paper on the proposals under consultation said this has resulted in variable safety and quality standards and no mechanism for enforcing regulations against vehicles licensed in a different local authority area.
To tackle that, the Welsh Government wants to set-up a national licensing authority with the job of taking over all existing tasks conducted by councils on taxis and private hire, with national standards for vehicles.
A second option could see national standards regulated by the 22 councils, with their enforcement powers extended, and with a mechanism for them to share information.
There are two types of licensed drivers - metered hackney cab drivers who can also do pre-booked work, and private hire drivers who do only pre-booked jobs.
The GMB union said the whitepaper did not tackle what has become known as "cross-bordering" - and believes that drivers should predominately work in the area they are licensed.
Regional organiser Lorraine Gaskell said: "We sat on one major junction in Cardiff and over the period of an hour we saw 44 licensed cars from Cardiff and 33 out-of-town taxis."
Paul O'Hara, a private hire driver from Cardiff who is part of a new driver-led co-operative called Drive, said many drivers are depressed with the amount of hours they have to work and the lack of family time.
He feared the new system would allow drivers from all over Wales to work anywhere they liked.
"The public deserve a good service but not to the detriment of drivers earnings, drivers health, because they have to stay out 12-13 hours a day to earn a living," said the driver, who is a GMB branch secretary.
"It's a crisis in the trade."
He said the white paper did not tackle the problem. A single authority "could see drivers from Anglesey getting a licence exactly the same as a driver from Cardiff, then working in Cardiff," he said.
His views were not shared by all Cardiff drivers.
David James, who drives for Uber, Ola and operators in Newport, has set-up a Facebook page backing cross-bordering.
The group, Taxis Without Borders, argues the main reason to continue cross-bordering is away from Cardiff.
In Monmouthshire "many of these villages the nearest town, and as such taxi company, will be in the neighbouring county, and it is these companies that they will rely on for transportation," he said.
TWB argued "closing borders will remove customer choice, as someone's preferred company may not be able to service the area they are in, despite being geographically local enough to be able to."
Mr James said customers who live outside of a city like Cardiff may be stuck only using local firms if cross-border work was restricted.
Allowing drivers to "work anywhere" gives customers a lot more choice, he said.
"Drivers should be allowed to work anywhere they want to. It's not just for the drivers benefits, it's for anyone's benefit," he added.
The Welsh Local Government Association said the consultation has been "light on detail", and councils felt they are the most "appropriate" organisations to do the job.
"Partnership working between key local agencies such as the police (including Transport Police), licensing service and the taxi trade offer the best opportunities for achieving sustainable late-night transport environments that also promote safety," said the body, which represents all local authorities in Wales.
A Welsh Government spokeswoman said it is their intention to move forward "in partnership" with councils, using their expertise and knowledge.
The spokeswoman added: "The Welsh Government is currently consulting on its proposals and we have already noted concerns raised in relation to both cross-border working and the numbers of licences being issued, which will be considered and will inform our next steps."