Immigration policy may be stuck between a "rock and a hard place" in dealing with public concerns, says an Oxford University report.
The study of 1,000 found people were most concerned about immigrant groups politicians could do little to cut.
The research by the university's Migration Observatory found broad overall support for cutting immigration to the UK, although less in Scotland.
The Home Office said its policies were in line with what the public wanted.
The Migration Observatory said it wanted answers to two questions that do not feature in standard opinion polls on immigration.
It asked respondents whom they referred to as immigrants and whether they wanted cuts to specific categories, such as asylum seekers, workers or students.
The report found approximately 70% of people want a cut in immigrants, broadly supporting previous surveys. A fifth said they thought immigration should stay at current levels.
Six out of 10 people thought the most likely reason someone came to the UK was for asylum, followed by just over half saying migrants mainly arrived to work.
This contrasted sharply with official statistics that show students make up the largest group of immigrants, followed by workers. Approximately 4% of all migrants in 2009 were asylum seekers.
This difference between actual and perceived trends could be seen when people described who they wanted to see cut.
Students came last on the list, but 56% of people wanted fewer asylum seekers.
The group that people were most concerned about was low-skilled workers.
The UK has a standing ban on unskilled workers from outside the EU, introduced by the previous Labour government, but cannot restrict the movement of EU citizens.
Scott Blinder, lead author of the report, said the team had tried to get behind blunt questions on immigration and establish whether public concerns mirrored government priorities.
He said that while the public wanted less immigration, a majority were concerned about the groups that it was most difficult for the government to reduce.
The coalition government has a target to cut net immigration to tens of thousands by the end of the Parliament.
It has introduced curbs on skilled workers from beyond Europe and students and it is also planning restrictions relating to family migration.
In response to the study, immigration minister Damian Green said: "We have made sweeping changes to get a grip on immigration in this country, closing down routes that were subject to abuse and taking action against those with no right to be here.
"This is clearly in line with what the public want us to be doing. There is much more to be done and we will stick to our course."
However, Dr Blinder said: "What this report shows very clearly is that the government is stuck between a rock and a hard-place.
"A clear majority of people in Britain would like immigration reduced, but they want the cuts to come from specific groups of immigrants, and these are often groups over whom the government has limited direct control, and sometimes groups that are comparatively small in number."
Ipsos Mori surveyed 1,002 people between 2 and 8 September for the Oxford study. Approximately 11% of those sampled were born abroad, 5% of them being British citizens.
Follow Dominic Casciani on Twitter.