David Cameron has defended plans to toughen welfare rules for EU migrants, saying he was sending a "clear message" to people that the UK was not a "soft touch" for claiming benefits.
He said he shared public concerns about the end of work restrictions on Bulgarians and Romanians next month.
Labour says he should have acted sooner and a European commissioner warned the UK risked being seen as "nasty".
But the prime minister said: "British people expect fairness."
"That is what this is about," he told BBC political editor Nick Robinson. "It is about fair treatment for people who work hard and do the right thing."
Mr Cameron has announced measures including:
- New migrants not getting out-of-work benefits for the first three months
- Payments being stopped after six months unless the claimant has a "genuine" chance of a job
- New migrants not being able to claim housing benefit immediately
- Deportation of those caught begging or sleeping rough, with no return within a year
- Quadrupling fines for employers not paying the minimum wage
Mr Cameron questioned the principle of free movement of people across the EU, saying this right could not be "unqualified".
"Yes, of course, there is a right to take up a specific position if you want to work but there should not be a freedom of movement to claim," he added.
He told the BBC controls were "not just aimed at Romanians and Bulgarians" but would apply to "anyone in other EU countries thinking of coming to Britain because it is easier to claim benefits".
"I think it is very important to send out a clear message that this is not the case."
Mr Cameron suggested a future Conservative government, as part of its pledge to renegotiate EU membership, could seek more control over migration policy.
Working with like-minded EU governments, he said, it would look at allowing member states to halt arrivals if numbers exceeded a set level.
He also suggested freedom of movement should only be fully allowed if the average income of a country's people was not too far below the EU average.
Transitional controls limiting Bulgarian and Romanian workers' access to the UK labour market - in place since the two countries joined the EU in 2007 - will expire at the end of the year.
There have been warnings of an "influx" of low-skilled workers and calls to review migrants' access to the NHS and welfare.
European Employment Commissioner Laszlo Andor described Mr Cameron's proposals as "an unfortunate over-reaction", adding that EU rules applied equally to all 28 member states and had been agreed to by the UK.
He told BBC Radio 4's Today programme the British public had "not been told all the truth" and that there were existing EU safeguards to prevent "benefit tourism".
Mr Andor said: "We would need a more accurate presentation of the reality, not under pressure, not under hysteria, as sometimes happens in the UK. I would insist on presenting the truth, not false assumptions."
Immigration from Poland and other countries had helped the UK economy, he said, arguing that the prime minister's suggestions risked "presenting the UK as a kind of nasty country in the European Union".
And Justice Commissioner Viviane Reding said: "If Britain wants to leave the single market, you should say so. But if Britain wants to stay a part of the single market, free movement applies. You cannot have your cake and eat it, Mr Cameron."
European Commission President Jose Manuel Barroso said he had spoken to Mr Cameron on Tuesday to remind him that free movement was a "fundamental" EU principle "that must be upheld".
He added: "There is clear evidence of its economic benefits but we are also aware of the challenges that this can also bring, particularly for local communities and services and EU rules already include measures to prevent abuse."
However, a Downing Street spokesman insisted the changes would happen "as quickly as possible", adding: "The prime minister is focused on ensuring we have the right rules."
'Still far too generous'
The Liberal Democrats said the proposed "sensible" changes would "restore confidence" in the immigration system and "ensure that the right to work does not automatically mean the right to claim".
But shadow home secretary Yvette Cooper said the prime minister was "playing catch-up" and copying a Labour idea.
During Commons exchanges with Home Secretary Theresa May, she said many of the measures would not be in place in time for the change in rules for Bulgarians and Romanians.
Many Conservative MPs urged Mrs May to defy the EU and extend the transitional controls on Bulgarians and Romanians, but she replied the UK would not go outside the law.
Downing Street says the rules allowing deportation of homeless people and six-month limit on getting jobless benefits would be in force from 1 January, as well as the tougher habitual residence test (which determines general eligibility to many UK benefits).
The three-month delay on claiming out-of-work benefits will take longer to implement because it requires legislation, while it is not yet known when, and for how long, EU migrants would not be able to claim housing benefits.
UK Independence Party leader Nigel Farage said the UK was "still being far too generous", adding: "Under his proposal, somebody can come here on 1 January from Romania and within 12 weeks be entitled to unemployment benefit."
MigrationWatch UK has said it expects 50,000 people to come from Bulgaria and Romania to the UK in each of the next five years but the Bulgarian ambassador has said he believes the figure will be much lower - predicting levels of about 8,000.