Ninety senior Conservative activists have urged David Cameron not to lift controls on Bulgarian and Romanian migrants working in the UK.
In a letter to the PM, they argue he could use a clause in EU law to prevent a "hugely disruptive and destabilising wave of mass immigration".
It allows countries to continue with controls if they have "serious labour market disturbances", they add.
They say the UK needs "space and time" to reduce long-term youth unemployment.
Conservative Justice Secretary Chris Grayling acknowledged there was "massive" public concern about the issue but suggested retaining the restrictions was not feasible as his party's Lib Dem coalition partners would not support the necessary measures in Parliament.
Bulgarians and Romanians gained the right to visa-free travel to the UK in 2007, when their countries joined the EU.
Since then, temporary restrictions have been in place meaning Romanians and Bulgarians have been able to work in the UK only if they are self-employed, have a job offer, or are filling specialist posts for which no British worker can be found.
These restrictions will be dropped on 1 January, having been extended to the maximum period of seven years.
Mr Cameron has said the government "can't stop these full transitional controls coming to an end".
But the letter, written by Conservative Grassroots chairman Robert Woollard and "backed by local association chairmen, former chairmen and other senior activists", says they "respectfully disagree" with the government's position.
It says a "safeguard clause" written into the EU accession treaty for the two countries "allows for the re-imposition of temporary restrictive measures in any member state if it is 'undergoing or foresees serious labour market disturbances'.
"Long-term UK youth unemployment - at 21% - is the third highest within EU and OECD [Organisation for Economic Co-operation and Development] countries," it adds.
The letter says it is therefore logical for the UK "to unilaterally exercise its opt-out on immigration matters under the Lisbon Treaty and extend the original restrictions to 2018 to allow the UK economy the space and time to reverse the long-term high youth unemployment trend".
"You must be aware that this is an untenable political position given the widespread opposition of the British people," it continues.
"It is also an unsustainable economic position in view of the huge pressure already placed on public services at a time when the country is still facing acute challenges within the economy."
The letter says: "How are local authorities going to be able to support unrestricted new immigrant individuals and entire families without additional financial support or increased local taxation?
"The fiscal position is simply untenable, irrational and grossly unfair - and may lead to social unrest."
Mr Woollard adds: "As a matter of urgency we urge you to call a special sitting of Parliament and bring forward a vote on Nigel Mills MP's proposed amendment to the Immigration Bill at the soonest."
More than 60 MPs have signed the Tory MP's amendment to extend the controls until 2018. But earlier this month, Commons Leader Andrew Lansley announced the bill would not be debated until next year - after the controls expired.
David Cameron has said the idea of freedom of movement within the EU, one of its most important principles, needs to be reconsidered as "massive" population shifts in the past decade have put pressure on countries across Europe.
Mr Grayling told the BBC that immigration controls were just one aspect of the UK's relationship with the EU that the Conservatives would change if they were governing on their own, instead of in coalition.
"David Cameron has said very clearly he would like to see tougher rules in future and we have always said we would have implemented tougher rules in the past," he told Radio 4's Today.
The Lib Dems and Labour were "happy with what we have got", he claimed, and only a majority Conservative government would be able to address issues "causing massive concern to the people of this country".
"We do not have a majority... the Conservative Party would not be able to get through the House of Commons some of the things that we would like to do in changing our relationship with the European Union."
Ministers have refused to predict how many migrants might arrive from Romania and Bulgaria.
With eight other EU countries - including France, Germany and Spain - lifting restrictions at the same time, they say such forecasts are difficult.
The government is tightening the rules on benefit claims by EU citizens who come to the UK and on Monday, it also announced that overseas visitors and migrants were to face new charges for some NHS services in England.
Meanwhile, a senior Romanian official has said he expects between 2,000 and 3,000 members of the minority Roma community to come to the UK after 1 January.