The European Union is a "force for social injustice" which backs "the haves rather than the have-nots", Iain Duncan Smith has said.
The ex-work and pensions secretary said "uncontrolled migration" drove down wages and increased the cost of living.
He appealed to people "who may have done OK from the EU" to "think about the people that haven't".
But Labour's Alan Johnson said the EU protected workers and stopped them from being "exploited".
The former Labour home secretary accused the Leave campaign of dismissing such protections as "red tape".
In other EU referendum campaign developments:
- Thirteen former US secretaries of state and defence and national security advisers, including Madeleine Albright and Leon Panetta, say in a letter to the Times that the UK's "place and influence" in the world would be diminished if it left the EU - and Europe would be "dangerously weakened"
- A British Chambers of Commerce survey suggests most business people back Remain but the gap with those backing Leave has narrowed.
- Five former heads of Nato claimed the UK would lose influence and "give succour to its enemies" by leaving the EU - claims dismissed as scaremongering by Boris Johnson
- Mr Corbyn is launching his party's battle bus, saying Labour votes will be crucial if the Remain side is to win
- The official Scottish campaign to keep the UK in the European Union is due to be launched in Edinburgh
Mr Duncan Smith's speech came after he told the Sun Germany had a "de facto veto" over David Cameron's EU renegotiations, with Angela Merkel blocking the PM's plans for an "emergency brake" on EU migration.
Downing Street said curbs it negotiated on in-work benefits for EU migrants were a "more effective" way forward.
In his speech in London, Mr Duncan Smith said EU migration caused a "downward pressure" on wages.
He singled out the Olympic Park in the capital, saying workers from Eastern Europe had undercut UK workers.
The document has moved here.
He warned of an "explosion of have-nots" and an increasing divide between "people who benefit from the immigration of cheap nannies and baristas and labourers - and people who can't find work because of uncontrolled immigration".
His speech was dismissed by Mr Johnson, who leads the Labour In campaign.
"For Iain Duncan Smith to suggest that those rights that actually help workers and stop them being exploited is part of the problem… many of the people in Iain Duncan Smith's camp call that red tape," he said.
"When they say they want to get rid of red tape they want to get rid of the right for part- timers to be paid the same as full-timers etc."
He also rejected the "haves and have-nots" argument, saying major trade unions were backing Remain because the EU had a "social dimension that's protected workers".
Mr Johnson also said it was an "extreme view" to believe there was "nothing right at all" about the EU.
Asked about "being called extremist", Mr Duncan Smith responded: "Those people in Remain really need to stop throwing threats and ridiculous terms like that around."
Launching Labour's EU referendum battle bus alongside Mr Johnson, party leader Jeremy Corbyn said immigration was "not necessarily" affecting wages or putting a strain on services.
"We actually don't need to start blaming people, we need to work together to deal with the issues of minimum wages and conditions," he added.
Mr Duncan Smith faced questions after his speech about his comments about the PM's renegotiation, telling reporters the EU was "absolutely point blank refusing to change what they do".
He echoed Boris Johnson in seeking to expose what he said was the gulf between the goals Mr Cameron set for himself in his "Bloomberg Speech" in January 2013 - when he announced plans for the referendum - and the reality of what he subsequently achieved.
In the Sun, Mr Duncan Smith, a former Conservative leader who resigned as work and pensions secretary last month in a dispute over disability benefit cuts, described the concessions gained as "very marginal" and suggested that, in return, the UK had lost its veto on future fiscal and political integration within the eurozone.
"The EU knew that our veto was very powerful and we have given it away," he told the newspaper.
"The reform failed. We got nothing on border control at all. We are now in a worse position than we were before.
"We have gone from wanting to lead in Europe to being on the end of a lead in Europe."
Analysis by Jenny Hill, BBC Berlin Correspondent
Mr Duncan Smith's remarks are considered preposterous in Berlin where there is a real sense of frustration at the direction in which the Brexit debate is heading.
The notion that Germany in effect sat on David Cameron and forced him to change his speech at the 11th hour is completely rejected here.
It's true that Germany wouldn't have supported an all-out brake on migration.
But that wouldn't have come as news to the British; Angela Merkel had always made clear she was unprepared to support any changes that undermined the European principle of freedom of movement.
For the Germans, this was always about compromise. German support was considered vital but, of course, David Cameron had 26 other member states to persuade and many here feel Mrs Merkel went out on a limb to help him with his renegotiation.
The Brexit referendum really matters in Berlin; Germany doesn't want to lose an important economic and ideological ally within Europe. And there is a degree of despair here at the tone of the debate in Britain.
Mr Duncan Smith suggested that Mr Cameron dropped calls for an emergency brake on all EU migration from a speech he gave in November 2014 - setting out in broad details his reform demands - amid German opposition.
"I saw the draft. I know that right up until the midnight hour, there was a strong line in there about restricting the flow of migrants from the European Union - an emergency brake on overall migration.
"That was dropped, literally the night before. And it was dropped because the Germans said if that is in the speech, we will have to attack it.
"The whole thing was shown to them. The Germans said from the outset, you are not getting border control. Full stop."
Emergency brake 'not most effective'
Mr Cameron told MPs last week that his renegotiation package - which included limits on access to tax credits and child benefits for new EU migrants, an opt-out for the UK from ever-closer union and safeguards for countries outside the eurozone - was substantial and an "additional reason" to stay in the EU but should not be taken in isolation when weighing up the broader benefits of EU membership.
But Mr Duncan Smith said the limits on in-work benefits would be "very complex" to implement and their impact would be limited as most EU migrants coming to the UK were doing so to find work not to claim benefits.
Responding to Mr Duncan Smith's claims, a Number 10 source said: "The prime minister made clear at the time that the government had looked at an emergency brake but he decided it was not the most effective way forward.
"That is why he decided to impose restrictions on benefits instead to end the something-for-nothing culture."
- Subscribe to the BBC News EU referendum email newsletter and get a weekly round-up of news, features and analysis on the campaign sent straight to your inbox.