The UK is working with several European governments to try to restrict the benefits migrants can claim when they move from one EU country to another.
Work and Pensions Secretary Iain Duncan Smith told the Sunday Times the UK, Italy, Germany, the Netherlands and Finland wanted to change EU law.
A three-month ban on EU migrants claiming UK out-of-work benefits came into force earlier this month.
But a senior EU official said migrants pay in more than they take out.
European commissioner Laszlo Andor told the BBC the UK risked "losing friends" and developing a bad image because of the way the debate on immigration was developing.
Deputy Prime Minister Nick Clegg, meanwhile, said it was "eminently sensible" to consider further changes to benefits for EU migrants.
But he cautioned changes must be done in conjunction with other European states or there would be a "danger" of tit-for-tat changes made by other governments.
"The idea that somehow we can apply new criteria to Germans, Fins, Dutch, Austrians you name it, but somehow no new conditions would apply to Brits living in other European Union countries is fanciful," he told Pienaar's Politics on BBC Radio 5 live.
'Committed to country'
The easing of restrictions on Bulgarians and Romanians working in the UK at the beginning of the year has seen the debate surrounding so-called benefit tourism resurface.
Mr Duncan Smith said there was "a growing groundswell of concern about the [immigration] issue" and Britain was "right in the middle of a large group of nations saying enough is enough".
He said he had been working with the other countries to bring pressure on Brussels to allow individual EU member countries to make their own rules stricter.
Mr Duncan Smith said Britain should ask migrants: "Demonstrate that you are committed to the country, that you are a resident and that you are here for a period of time and you are generally taking work and that you are contributing."
He added: "At that particular point... it could be a year, it could be two years, after that, then we will consider you a resident of the UK and be happy to pay you benefits."
Sources close to Mr Duncan Smith stressed he was expressing an aspiration for the future rather than spelling out a policy.
It comes after Europe's Employment and Social Affairs Commissioner Mr Andor insisted migrant workers were net contributors to the UK economy.
"We shouldn't assume that the UK welfare system is a lot more generous than that of many other countries," he told BBC Radio 4's The World this Weekend programme.
"Migrant workers altogether are net contributors to the system. They take out much less in the form of benefits or welfare services than what they contribute in the form of taxes or contributions to the system."
His remarks echoed that of his EU colleague Viviane Reding, vice-president of the European Commission, who said last week that it was a "myth" to speak about an "invasion of foreigners" stealing jobs and draining welfare and health resources.
Christian Dustmann, an economics professor at University College London who has published research on how much migrants claim in benefits in comparison to people born in the UK, told BBC Radio 5 live on Sunday there was clear evidence about who was claiming more.
"We have looked at the overall receipt of transfers and benefits, which of course include child benefit, housing benefit and other forms of benefits, and what we find is that migrants from EU countries are 33% less likely than UK natives to claim any form of benefits," he said.
Prof Dustmann said there was "very little concern that immigrants from EU countries are free-riding on the UK's welfare system".
Matthew Pollard, executive director of Migration Watch UK, a think tank that supports tighter immigration controls, accepted EU migrants claimed less than UK nationals in out-of-work benefits but said it was "still right for the government to restrict access".
"It goes against people's sense of fairness that the EU want an EU migrant to be treated in exactly the same way as a Brit when it comes to out-of-work benefits. This undermines confidence in the welfare system as well as the EU in general," he told 5 live.
He said migrants claimed more in terms of "in work" benefits, such as working tax credits and housing benefit, and there was no economic case for mass immigration.
Last week UKIP leader Nigel Farage called for migrants to be barred from receiving benefits until they have been living in the UK for five years.
London Mayor Boris Johnson suggested any ban should be two years.
Labour has said it supports the government's three-month ban, which it said was "reasonable and achievable".
Meanwhile, more than 90 Conservative MPs have written to David Cameron urging him to give Parliament a national veto over current and future EU laws.