People not having much money to spend is the main reason high streets struggle, according to researchers.
Tactics such as lowering business rates or taxing online sales were unlikely to help because they ignore the problem of a weak local economy, the think-tank Centre for Cities said.
It described Swansea and Newport as weak cities and Cardiff as strong.
Stronger cities have more businesses that export and need higher-skilled staff, bringing wealth into the area.
In Cardiff, 22% of jobs are in these high-skilled exporting firms, compared to 19% for Newport and 16% for Swansea.
Any government funds to support High Streets should be used to develop a higher-skilled workforce, better quality offices and improved transport, researchers argued, saying this would make the most difference because it helped attract businesses that create well-paid jobs.
High streets of the strongest cities are more varied and less reliant on shops, with nearly half of their city centre businesses selling food and drink, and with more amenities such as cinemas and venues for the performing arts.
In Newport, the level of places where you can eat is below 10%.
The future of Swansea
In Swansea, the average household disposable income is £14,908, about three-quarters of the UK average.
The city has had various setbacks, such as the cancellation of the electrification of the rail line from Cardiff, and the council is now driving the tidal lagoon energy project after the UK government decided not to back it.
There have also been rows over the Swansea Bay City deal.
But it also has two universities which have brought forward expansion plans and there is widespread regeneration under way and more in the pipeline.
This is "to create new leisure, office and housing that will see thousands more people live and work in the city centre and bring high-tech professional jobs to the area", the council said.
It will take many years to assess whether this investment pays off and creates the hoped-for wealth.
The report comes as research suggested cities in Wales were some of the hardest hit by the closure of chain retailers.
It said internet competition was not the primary cause of High Street decline and city centres offering consumers something they were unable to get online were more likely to succeed in the long-term.
On the flip side, it said cities with stronger economies could become unaffordable and warned planning rules must enable them to build so there is enough property to meet demand.
Centre for Cities' chief executive, Andrew Carter, said good jobs and a strong local economy were the key.
He said: "Any interventions that seek to improve cities' amenities without boosting consumer spending power are doomed to fail from an economic perspective.
"Policy should focus on improving cities' overall economic performance."