Europeans will have to prove they are "genuinely seeking employment" to claim UK jobless benefits for more than six months, David Cameron has said.
The prime minister said it was among measures to ensure people came to the UK "for the right reasons" after it became a "soft touch" under Labour.
But Bulgaria's UK ambassador said the UK's rules were already seen in his country as "very restrictive".
Labour warned against an "arms race on immigration rhetoric".
Migrants from the European Economic Area - the EU member states plus Iceland, Liechtenstein and Norway - currently have to show they have a "reasonable chance" of finding a job to receive unemployment benefit for more than six months.
Downing Street said they would now face a more rigorous test to assess whether they had a "realistic prospect" of getting a job, with the ability to speak English one of the criteria.
In his speech in Ipswich, the prime minister said there were "concerns, deeply held, that some people might be able to come and take advantage of our generosity without making a proper contribution to our country".
"These concerns are not just legitimate; they are right and it is a fundamental duty of every mainstream politician to address them."
No 10 was unable to give any figures on the scale, cost and numbers of so-called benefit tourists, although Department for Work and Pensions figures suggest 17% of working-age UK nationals claim a benefit, compared with 7% of working age non-UK nationals.
Transitional restrictions on Bulgarians and Romanians working in the UK are due to be relaxed next year.
Since the countries joined the European Union in 2007, their peoples have been able to come to the UK to live and have been able to take jobs either via a work permit system, or by being self-employed, or in a variety of jobs from domestic work to seasonal agriculture.
According to the Office for National Statistics, in July 2012 there were 94,000 Romanians and 47,000 Bulgarians resident in the UK.
The end of existing controls will give those who want to work in the UK the same rights for welfare and NHS care as foreign nationals from the other 24 EU nations.
Mr Cameron said: "We can't stop these full transitional controls coming to an end. But what we can do, is make sure that those who come here from the EU - or further afield - do so for the right reasons: that they come here because they want to contribute to our country, not because they are drawn by the attractiveness of our benefits system, or by the opportunity to use our public services."
The prime minister added: "Under the last government immigration in this country was too high and out of control. Put simply, Britain was a soft touch."
He said immigrants in future would be "subject to full conditionality and work search requirements and you will have to show you are genuinely seeking employment - if you fail that test, you will lose your benefit".
He said: "And as a migrant, we're only going to give you six months to be a jobseeker. After that benefits will be cut off unless you really can prove not just that you are genuinely seeking employment but also that you have a genuine chance of getting a job.
"We're going to make that assessment a real and robust one and, yes, it's going to include whether your ability to speak English is a barrier to work."
Bulgaria's London ambassador, Konstantin Dimitrov, told the BBC that trying to put figures on the number of his compatriots who might migrate when restrictions are eased was "irrelevant", as "most of the Bulgarians who wanted to find work in the UK have already done so".
He added: "Contrary to the prevalent opinion here about the UK being seen as a soft touch, your system is seen by Bulgarians as very restrictive."
Mr Dimitrov also said: "Luckily Bulgarians don't believe in sham marriages... Nobody need worry about a possible influx of undesirable Bulgarians."
In his speech, Mr Cameron said changes to health care would be introduced, with the UK getting "better" at "reciprocal charging", charging foreign governments for treatment provided to non-working overseas nationals.
He added that action was being taken to tackle illegal immigration, including making private landlords responsible for making sure their tenants were legally in the UK - and facing fines if it turned out they were not.
Immigrants would be kept off council house waiting lists in England for at least two years, under plans for councils to introduce a residency test.
Councils can already set their own criteria, but many do not.
Mr Cameron said: "We can not have a culture of something for nothing. New migrants should not expect to be given a home on arrival."
But the Local Government Association said it was "very concerned", and councils should decide how to meet housing need.
Jonathan Portes, director of the National Institute of Economic and Social Research, told the BBC that people "coming from outside the UK, and especially people coming from outside the European Union, are significantly less likely than British nationals, and people born here, to claim benefits".
For Labour, shadow home secretary Yvette Cooper said: "It's right to have conditions on benefits and public services for immigration. Most people who come to this country work and contribute, but there are restrictions because the system needs to be fair and seen to be fair.
"However, the government's proposals announced today seem to be very confused and are unravelling. And at the same time there was no significant action to tackle illegal immigration or labour market exploitation which we know have been getting worse.
"We won't support an arms race on immigration rhetoric, we want practical and sensible measures that make the system work."
On Friday Lib Dem leader and Deputy Prime Minister Nick Clegg gave a speech on immigration in which he called for £1,000 deposits to be demanded for visa applicants from "high-risk" countries, with the money repaid when they leave the UK.