Michael Gove says leaving the EU is necessary for people to support Britain's "multi-racial, multi ethnic society".
The pro-Leave justice secretary said the government could not limit the number of arrivals from inside the EU.
On a Question Time special, a Spanish woman who works in the UK criticised him and said: "We are not the enemy".
Mr Gove also attacked chancellor George Osborne's "Brexit Budget" warning of emergency spending cuts.
A letter claiming the chancellor's position would be "untenable" if he tried to cut NHS, police and school spending has been signed by 65 MPs.
Mr Gove said he would not back such a Budget and criticised "dire warnings" coming from the Remain campaign.
"I think it's a shame that the Remain camp are talking this country down," he said.
In other referendum news:
- Home Secretary Theresa May says the government "should look at further reform" of free movement of people, if the UK votes Remain.
- Tory backbenchers have reacted angrily to George Osborne's warning of cuts to public spending and tax increases in the wake of a vote to leave the EU
- Nigel Farage and Sir Bob Geldof, leading rival Leave and Remain flotillas, trade verbal blows on the Thames.
- The CBI said business groups from countries including Norway, Switzerland and Canada had outlined the "serious shortcomings" in their alternative relationships with the EU.
- Rolls-Royce writes to its staff saying it backs a Remain vote.
- Scotland's First Minister Nicola Sturgeon warned that the UK voting to leave the European Union could present the risk of a "right-wing Tory takeover".
Prime Minister David Cameron will make the case to remain, in a second Question Time special on Sunday.
During his 45-minute grilling, the justice secretary was challenged by audience members over warnings from financial institutions and other bodies about life outside the EU.
Responding to a woman who said leaving would make it more difficult to trade with other EU countries which are the "lifeblood" of her business, he said other countries would "take leave of their senses" if they chose not to continue to trade with the UK.
He conceded there would be "bumps in the road" if voters back Brexit, but said the UK "will be in a better position to deal with them".
He said: "My view is that whatever happens in the future we will be in a strong position to deal with any crises that occur as a result of leaving the EU."
Mr Gove said his father had to close his fish merchant business because of EU policy. And he criticised The Guardian after it said his father had contradicted his version.
"My dad was rung up by a reporter from the Guardian who tried to put words into his mouth," he said, adding that his father was "clear... that the business he invested so much care and time in had to close as a result of the Common Fisheries Policy".
On immigration, Mr Gove said if there is a vote to leave, the government would "bring down the numbers" in the Parliament after 2020, by which time the UK's exit would have been completed.
'Chuck us out'
A Spanish audience member - who is unable to vote on 23 June - said she had lived in England for 14 years and did not feel "welcome" as a result of Mr Gove's call for a crackdown on EU migration.
"You use us to your convenience, and when we are no use to you any more you chuck us out," she told him.
Mr Gove said he valued her contribution but that it was "undeniable" that "to continue to have support for migration we need to be able to control the numbers".
Working people's wages were "held down" by immigration, he said, calling for the UK to "take back control".
The Question Time programme is the latest in a series of special referendum shows being broadcast on the BBC.
Campaigners from both sides will take each other on in a live debate at Wembley Arena on 21 June. The line-up has yet to be announced.