Nick Clegg has said he was "very surprised" to see Home Office vans "driving aimlessly around North London" telling illegal immigrants to go home.
He said the vans, with the slogan "go home or face arrest", were not a "very clever way" of tackling the issue.
Mr Clegg told BBC Radio 5 live no Liberal Democrats - including Home Office Minister Jeremy Browne - knew about the pilot scheme in advance.
Downing Street had said the vans were approved by the "Home Office team".
No 10 also said on Monday that the pilot scheme using the vans had worked, although the Home Office said it was too soon to make a final assessment as a poster and leaflet campaign was continuing.
Mr Clegg said: "Unless there is overwhelming evidence that this is a really effective way of bolstering public confidence in the immigration system and bearing down on illegal behaviour in the immigration system, I'm going to need a lot of persuasion.
"What are we going to have next? Home Office vans driving around saying, 'Please don't shoplift,' or, 'Please don't steal this car,'?"
During the BBC radio phone-in Mr Clegg said: "I think people should play by the rules and play by the law. I want to see us make sure that the public have confidence in the immigration system, that it works properly.
"I don't happen to think that having a couple of vans driving around north London is the way of actually inspiring public confidence that we have an immigration system that is working properly.
"Tone is important. I think you can enforce the law effectively without instilling a tone which is unsettling to communities, particularly to mixed communities.
"What I want to see the Home Office concentrate on instead is making sure that we have a simple system, that's done in many other countries, where you count people in and you count people out."
He added: "I said to the Home Office in very clear terms, I was talking to the immigration minister yesterday - he's a very good guy, by the way, Mark Harper - I was saying the effort that should be spent is not on vans drifting around north London but on actually reinstating exit checks."
Mr Clegg said he thought the exit checks were "wrongly removed in the past" and the government was committed "on my personal insistence... to reinstating [them] as part of our coalition agreement".
Mr Clegg said it was "frustrating" that the Home Office "has not yet made as much progress on this as they should do under the terms of our coalition agreement".
He added that "the only way you can deal with this problem of people overstaying their visa is if we have a system which logs when people's visas have expired and whether they have crossed the border and left this country".
"I will now redouble my efforts to ensure the Home Office really does its homework and reinserts those exit checks at the earliest possible opportunity," Mr Clegg said.
The row over the vans - which were called "stupid" by Lib Dem Business Secretary Vince Cable - is the latest sign of divisions between the two coalition parties.
Mr Clegg said: "You are going to have to ask a Conservative politician why the Conservative Party three years ago presented itself as green and compassionate and a centre-ground party and now appears to be resorting to some more traditional Conservative right-of-centre signature tunes on Europe, immigration, welfare.
"Important though those issues are, I remain of the view as a centrist politician, as someone who is determined to anchor the government in the centre ground, that's one of the roles of the Liberal Democrats in the government, that actually the vast majority of decent people in this country are in the centre ground and want to see politicians governing from the centre, not from either the right or left."
Ahead of Mr Clegg's comments, Conservative Immigration Minister Mark Harper wrote in the Daily Mail that the posters were not racist, but were designed to inform people that the government would help people here illegally to return home.
"This government is successfully controlling immigration, while continuing to ensure that Britain attracts the brightest and the best from around the world," he wrote.
"But illegal immigration must be tackled. If the poster campaign helps with that, why would anyone be opposed to it?"
For Labour, shadow Home Office minister Baroness Angela Smith said: "These ill-judged ad vans are cynical and stupid politics from a government not getting the basics right on immigration.
"Nick Clegg has now admitted the Home Office have failed to make much progress on the basics, yet he has added to the chaos and confusion within government over the ill-judged ads. If he opposes these ad vans and has spoken to the immigration minister, will they continue? And did his own Lib Dem Minister in the Home Office sign them off?"