Leading Leave campaigner Michael Gove says he "shuddered" when he saw UKIP's "breaking point" anti-EU poster showing a long queue of migrants.
The justice secretary joined a number of politicians attacking the poster, featuring a photo taken in Slovenia, with George Osborne saying it had "echoes" of 1930s' literature.
But UKIP leader Nigel Farage defended it, saying it was "the truth".
He said it was "about us being part of a European Union that isn't working".
Critics say the migrants shown in the poster were not coming to the UK and have accused UKIP of trying to scare voters ahead of Thursday's EU referendum.
Mr Gove said: "When I saw that poster, I shuddered. I thought it was the wrong thing to do."
But Mr Farage, who is not part of the official Leave campaign, defended it in a series of interviews on Sunday morning, saying the scene depicted was "a direct result" of German Chancellor Angela Merkel's decision to open her country's doors to Syrian refugees, which he described as "one of the biggest political failures of modern times".
"The point of that poster was to say that Europe isn't working," he told Pienaar's Politics on BBC Radio 5 live, adding: "Something that's true can't be a scare."
Mr Farage said anyone who "dares challenge" the "narrow Westminster consensus" on free movement attracts abuse, and said his party was trying to put across a "positive message".
He told Murnaghan on Sky News: "It was a one-day poster. I have got six posters and you will see the next one tomorrow morning in the national press and online.
UKIP's next posters would be "all about this country not the wider point it was making there about the European Union failing us all," he added.
Asked whether he was stoking up hatred, he told ITV's Peston on Sunday: "I think I have been a politician who has been a victim of it, to be honest with you.
"When you challenge the establishment in this country, they come after you, they call you all sorts of things."
Mr Farage said his side of the EU debate had the "momentum" until the death of Labour MP Jo Cox, which prompted both campaigns to be suspended.
"It has had an impact on the whole campaign for everybody," Mr Farage said.
His party's poster has been attacked by politicians on both sides of the EU debate.
Pro-Leave Commons Leader Chris Grayling said: "It's the wrong poster, it was the wrong approach, it's the wrong view."
While on the Remain side, Mr Osborne described it as "disgusting and vile" with "echoes of literature used in the 1930s", while Scotland's First Minister Nicola Sturgeon called for it to be withdrawn.
"My argument is let's deal with the impact, let's invest in housing and public services rather than blame immigrants or take a view that we should somehow close our borders," she said.
Labour leader Jeremy Corbyn said it was an "appalling" poster showing "a lot of desperate people fleeing from war".