David Cameron has rejected the idea he is "scaremongering" by claiming migrant camps like the "Jungle" in Calais could move to England if the UK left the EU.
The PM said a vote to leave would give French politicians the chance to "tear up" the deal which lets UK border guards check passports at Calais.
Campaigners for the UK to leave the EU say the deal is between the two countries and not an EU deal.
UKIP said the PM's claim was "based on fear, negativity and falsehood".
The 13-year agreement with France had "nothing to do with our EU membership," said Vote Leave, while Leave.EU said there was "no reason for this to change on leaving the EU".
Mr Cameron's former defence secretary Liam Fox said he was "sad and disappointed to see our prime minister stoop to this level of scaremongering", while fellow Conservative MP Sarah Wollaston said Mr Cameron was "taking voters for fools".
An in-out referendum on EU membership has been promised by the end of 2017, although it is widely expected to be at the end of June, if Mr Cameron can secure agreement to his reform proposals at the next summit of EU leaders on 18 February.
About 5,000 migrants and refugees trying to reach the UK are thought to live at the "Jungle" camp in Calais.
Under the terms of the February 2003 Le Touquet treaty between the UK and France, checks for migrants trying to illegally stow away on lorries or trains heading for Britain are carried out by British border guards at a number of French ports.
Under the system of "juxtaposed controls", which was transposed into UK law in June 2003, the UK operates immigration checkpoints in Calais and Dunkirk, while France has a checkpoint at Dover.
Analysis: By BBC assistant political editor Norman Smith
Asked about the suggestion that he was "scaremongering" by linking a vote to leave the EU with the moving of the Calais migrant camps to the south coast of England, Mr Cameron said it was a very real risk.
"There are any number of opposition politicians in France who would love to tear up the excellent agreement we have with France to make sure that we have our borders on their side of the Channel - I don't think we should give those politicians any excuse to do that," he said.
"If we can get this deal in Europe, if we can get this renegotiation fixed and we can stay in a reformed Europe, you know what you get. You know the borders stay in Calais."
The deputy Mayor of Calais, Philippe Mignonet, bolstered his case by telling the BBC: "We will have to cancel these agreements, because England won't be in Europe anymore. So that will really be a foreign country for us."
But the UK's former chief inspector of borders and immigration, John Vine, told the BBC's World at One he didn't think the UK's exit from the EU would lead to an immediate collapse in cross border arrangements with France.
"I don't necessarily think this would collapse overnight. If it didn't exist there would still be border controls. We would have to rely on the Belgians and the French," he said.
Mr Vine pointed out asylum seekers could only claim asylum once they were on British soil.
What are the current arrangements with France?
The 'Treaty of Le Touquet' is an agreement between France and the UK that has been in place since 2003 and governs border control arrangements and immigration.
The treaty ensures that immigration checks are carried out before passengers embark on cross channel services - French Border Police have immigration checkpoints at Dover while the UK has immigration checkpoints at Calais and Dunkirk. But at Calais and Dunkirk, passengers also go through French exit checks, as well as UK immigration entry checks.
In theory this stops those seeking to reach the UK from doing so without their immigration status being checked first. But this has also led to the establishment of camps in Calais such as the so-called Jungle and previously, Sangatte.
The BBC's political correspondent Chris Mason said Mr Cameron's warning was a sign that Downing Street was planning to emphasise what it sees as the important contribution the EU makes to the country's national security.
But Vote Leave said Mr Cameron's claims had no basis in reality. "UK border controls are in France because of a bilateral treaty, not because of our EU membership, and a result of the camps in Calais, not the cause of them," said Matthew Elliott, the campaign group's chief executive.
"Clearly, No 10 is in a blind panic over the failing renegotiation."
Leave.EU co-founder, Arron Banks said: "The agreement we have to process migrants in Calais is with France, not the EU. There is no reason for this to change on Leaving the EU.
"The key issue here is that we should not have to accept migrants that have arrived from a safe country. The scandal of countries effectively waving migrants through Western Europe to our shores will only end when we are able to control our borders again."
Further reading on the UK's EU referendum
EU renegotiation: Did Cameron get what he wanted?
Referendum timeline: What will happen when?
The view from Europe: What's in it for the others?
Mr Cameron has been touring European capitals trying to build support for his draft renegotiation package, which he has said is the basis for a new settlement for Britain in the EU but which has been dismissed as inconsequential by those wanting to leave the EU.
Mayor of London Boris Johnson, a key figure in the Conservative Party, has said the case for staying or leaving is "finely balanced" and he would only decide which way to campaign when the terms of the final agreement were known.
"This is the moment to stiffen the sinews, summon up the blood, squint down the barrel and only when you see the whites of their eyes should you finally let fly and decide whether to stay or leave the EU," he wrote in the Daily Telegraph.