Men with fewer qualifications in manual jobs are more likely to be hit by post-Brexit trade barriers, according to the Institute for Fiscal Studies (IFS).
Factory workers in industries that sell a large amount to the EU are particularly at risk, the IFS warned.
A number of studies have indicated that people with lower or no qualifications were more likely to vote Leave.
A government spokesperson said it had proposed frictionless trade between the UK and the EU.
"We have put forward an ambitious proposal that will allow frictionless trade between the UK and the EU, protect jobs across all sectors and deliver a successful Brexit that works in the interests of the whole UK," the spokesperson said.
"It is in no one's interests for there to be trade barriers between the UK and the EU which is why we have proposed a UK-EU free trade area underpinned by a common rulebook for goods."
The UK has extensive trade links with the European Union, with 44% of UK exports destined for the bloc, and more than half of imports, the IFS said.
Leaving the single market and customs union "will increase trade barriers and make both importing and exporting more costly", it said.
The most exposed group to increased trade barriers after Brexit are men who work in car manufacturing, clothing, chemicals, and pharmaceuticals factories who have GCSE qualifications or below.
These industries sell a large amount of products to the EU. The car industry is particularly at risk, because it also imports parts used in vehicle manufacture from the bloc.
The IFS report comes a day after Nissan warned that a no-deal Brexit would have "serious implications for British industry".
Nissan is a major employer in Sunderland, where it has a plant. Sunderland was one area that voted heavily in favour of Leave in the EU referendum.
About 3.7 million people work in industries the IFS regards as "highly exposed", meaning they would lose more than 5% of their value if trade was under WTO rules after Brexit.
There are also "very highly exposed industries" that employ about 500,000 process, plant and machine operatives.
These people "may find it particularly hard to move into less exposed industries as workers in this group tend to be older and less likely to have formal qualifications than other workers", said IFS research economist Agnes Norris Keiller.
Data from the BBC indicated that older people were more likely to vote Leave in the referendum.
Workers in Northern Ireland and the West Midlands could be hit particularly hard, as about a quarter of men with fewer qualifications work in highly exposed industries in these areas, as opposed to about a fifth nationally.
However, other sectors are also at risk, the IFS said. The financial services sector is particularly exposed, as about 12% of its business is with the EU.
"Crucial here will be the scale of non-tariff barriers, including the potential loss of passporting rights, after Brexit," the IFS said.
Agricultural workers may benefit from Brexit trade if consumers buy more British produce as EU food imports become more expensive.
However, the industries that stand to gain make up only a small part of the UK economy, the IFS said.
Estimates from Ipsos Mori indicated that people with fewer qualifications tended to vote Leave in the 2016 EU referendum.
Data obtained by the BBC showed that populations with lower qualifications were significantly more likely to vote Leave.
And the LSE said in October last year that people qualified to A-level, GCSE level, or with no qualifications were more likely to vote for Brexit, while people with university and post-graduate degrees were more likely to vote Remain.