A former chairman of the Conservative Party has switched her support from the Leave campaign to Remain.
Baroness Warsi told the Times the "hate and xenophobia" of the Leave campaign was "a step too far".
She said she realised she could no longer support Leave when she saw UKIP's "breaking point" anti-EU poster.
Vote Leave said it was not aware Lady Warsi had joined its campaign, while Nigel Farage defended the poster, saying it was "the truth".
A number of politicians from both sides of the EU debate accused UKIP leader Mr Farage, who is not part of the official Leave campaign, of trying to scare voters ahead of 23 June's referendum after he unveiled a poster showing a long queue of migrants.
Leading Leave campaigner Michael Gove said he "shuddered" when he saw the photo, which was taken in Slovenia, with George Osborne saying it had "echoes" of 1930s' literature.
Lady Warsi, who resigned from the government in 2014 over its position on Gaza, told the Times: "That 'breaking point' poster really was - for me - the breaking point to say, 'I can't go on supporting this'.
Speaking on BBC Radio 4 Today's programme Lady Warsi addressed criticism that she had never been part of the Leave campaign.
"I was making the case to leave long before Vote Leave had been established, " she said.
She said she had a discussion with senior Leave campaigners in which she set out her "optimistic" vision for Britain, which "trades freely, open to the brightest and best and rooted in its humanitarian instinct".
"Unfortunately what we are seeing as a vision for Britain are lies and xenophobic campaigning. Why is it people like me, who are instinctively Eurosceptic, are feeling they need to leave Leave?
"Because day after day, what are we hearing? The refugees are coming, the rapists are coming, the Turks are coming."
Speaking about her decision to switch sides, she told the Times: "It's not an easy decision - this has been a difficult personal journey.
"When I look at the people who are now saying the things they are saying and the people who are supporting that approach, the BNP, Donald Trump, Marine le Pen, Austria's Freedom Party - every day it feels like the far right is coming out to stand by Leave."
However, Leave campaigners have said they never knew Lady Warsi was a part of their campaign.
Vote Leave tweeted: "Genuinely puzzled by Sayeeda 'defection'. We weren't aware she was ever part of the #VoteLeave campaign..."
Employment minister and Vote Leave campaigner Priti Patel said Lady Warsi was "just one voice" and pointed out Vote Leave had had people join its campaign from Remain.
"We are seeing a range of voices during this debate. Only at the weekend Lord Guthrie, who famously came out to Remain, and now he has come over to the Vote Leave case for a range of reasons."
Meanwhile, Conservative MEP Daniel Hannan tweeted: "When I invited Sayeeda Warsi to join the Leave campaign, she declined. Fair enough, obviously. But how is this a "defection"?
Mr Farage defended the breaking point poster in a series of interviews on Sunday - and again in a BBC interview with Radio 4's Today programme on Monday.
He said it highlighted "one of the most irresponsible policy decisions of modern times" by German Chancellor Angela Merkel to accept hundreds of thousands of migrants, which he said had led to the EU's border-free Schengen area "breaking up".
"I didn't invent that picture. The picture was real, the picture was on the front pages of all our national press last year," the UKIP leader said.