The UK is to halve to three months the time EU migrants without realistic job prospects can claim benefits.
David Cameron said the "magnetic pull" of UK benefits had to be addressed so people came for the right reasons and the rules "put Britain first".
The prime minister also warned people who are in the UK illegally: "We will find you, we will send you home."
Labour said the PM's rhetoric masked a record of "failure" on immigration and "firm action" was needed.
Ministers have been reluctant to say how many people are likely to be affected by the benefit curbs but have claimed the changes could save £500m over five years.
But the Office for Budget Responsibility distanced itself from the £500m figure, saying it was an estimate that related to a wider package of benefit curbs and compliance checks announced in March's Budget.
The independent watchdog said it had not discussed the latest proposal with ministers or attempted to calculate its likely impact.
The BBC News Channel's chief political correspondent Norman Smith said it appeared the announcements were more about sending a message to UKIP voters and showing that the government was serious about tackling immigration, than saving money.
European Union officials responded to the plans by saying there was no evidence to show migrants moved to Britain to claim benefits.
"(Migrant workers are) of immense economic benefit to the member states in terms, in particular of responding to skills gaps and labour shortages," European Commission spokesman Jonathan Todd said when asked about the prime minister's announcement.
The Commission said it would scrutinise the new measures carefully to ensure they complied with EU law.
On a visit to see immigration officers at work, Mr Cameron said: "We want an immigration system that puts Britain first so if you come here illegally... we will make it harder for you to have a home, to get a car, to get a job, to get a bank account and when we find you - and we will find you - we'll make sure you are sent back to the country you came from."
He unveiled details of the plans to curb the rights to claim some benefits for some EU nationals who are legally in the UK in a Daily Telegraph article on Tuesday.
He said the last Labour government presided over a "no-questions-asked" welfare system that "drew migrants to the country for the wrong reasons".
He cited other measures recently coming into force to tackle abuses - such as new powers revoking the driving licences of those not entitled to be in the country - as evidence that the government was building "an immigration system that puts Britain first".
"We are making sure that people come for the right reasons - which has meant addressing the magnetic pull of Britain's benefits system," he said.
Crunching the numbers, by Anthony Reuben
How many people will be affected by this change of policy?
To get an idea of this figure, let's start with the maximum possible number, which is 60,100. That's the latest figure (February 2013) from the Department for Work and Pensions for the number of people claiming JSA who were EU nationals from outside the UK at the time when they applied for their National Insurance card. That's 5.8% of all claimants.
We are only interested in those who have been claiming JSA for between three and six months because EU migrants are already unable to claim for more than six months and they will still be able to claim for three.
The latest ONS figures tell us that of those 1.04 million total claimants, 179,500 of them, about 17%, had been claiming for between three and six months.
That's for the whole population, but it would be reasonable to assume that EU migrants do not claim for longer than the population as a whole because they have extra incentives to get a job as a result of the threat to stop their benefits.
So the number of people affected is unlikely to be above 10,000.
Next, you have to subtract the people who have made National Insurance contributions and those who, in the prime minister's words, "had very clear job prospects".
We have no idea how much this policy will save. The figure of £500m has been trumpeted around as being the Office for Budget Responsibility's estimate of the saving over five years.
But the OBR has now said that figure was a valuation for various measures in Budget 2014, and nothing to do with today's announcements.
"We change the rules so that no-one can come to this country and expect to get out-of-work benefits immediately - they must wait at least three months.
"And we are announcing today that we are cutting the time people can claim these benefits for. It used to be that European jobseekers could claim JSA or child benefit for a maximum of six months before their benefits would be cut off, unless they had very clear job prospects.
"We will be reducing that cut-off point to three months, saying very clearly you cannot expect to come to Britain and get something for nothing."
The government has been steadily tightening the criteria under which EU migrants are eligible to claim benefits after coming under political pressure.
In January, it announced that EU migrants would not be able to claim out-of-work benefits until three months after arriving in the UK and would only be eligible for jobseeker's allowance for six months unless they have genuine prospects of finding work.
Mr Cameron announced that this time limit - which also applies to child tax credit and child benefit - will be halved to three months from November.
Shadow home secretary Yvette Cooper said the government had failed to take "firm action" to address the issue.
"Behind the rhetoric the true picture of this government on immigration is one of failure, with net migration going up, despite David Cameron's promise to get it down to the tens of thousands," she said.
"The government should get a grip and finally implement Labour's proposals to stop the undercutting of wages and jobs for local workers by the exploitation of low-skilled migrant labour, including banning recruitment agencies that only hire foreign workers and pressing for stronger controls in Europe."
The UK Independence Party's migration spokesman Steven Woolfe called the PM's announcement "a cynical and vacuous attempt to fool the British public".
He said: "Britain is crying out for an Australian-style points system for migration, so that we can judge potential migrants from all over the world on the same basis: whether they can make a positive contribution to life in our already-overcrowded country.
"We will never be able to introduce such a system while we are locked in the EU."