It is "outrageous" to describe people who are worried about the impact of migration as racist, Archbishop of Canterbury Justin Welby has said.
There was "genuine fear" of the impact on housing, jobs and the NHS, he told Parliament's The House magazine.
But he said Britain's pledge to take 20,000 Syrian refugees did not compare well with the efforts made by Germany.
He also called for a "visionary debate" on the EU referendum and said there was no "correct Christian view" on it.
The archbishop said he felt the referendum debate was not yet addressing questions about the UK's role in the world.
Archbishop Welby called on those who want to leave the EU to set out what Britain would be like after leaving in a variety of respects, including its international "attitude" and its values.
"And from those who want to stay, how would we change the European Union? How would we make it more effective if we remained in it? What's our vision?" he said.
The UK will vote on whether to remain in the EU or leave on 23 June.
The archbishop praised the "absolutely superb" humanitarian work Britain was doing in refugee camps surrounding Syria and Iraq.
However, he said Britain's pledge to accept 20,000 Syrian refugees by 2020 did not compare well with the number accepted by Germany and urged the UK to take "our share".
"I was in Berlin, and the churches there are doing the most extraordinary things, as are the German people," the archbishop said. "They took 1.1 million last year. And it does make 20,000 over several years sound really very thin."
He recognised that some people had concerns about the pressure that new arrivals put on communities and services. The archbishop said: "There is a tendency to say 'those people are racist', which is just outrageous, absolutely outrageous.
"Fear is a valid emotion at a time of such colossal crisis.
"This is one of the greatest movements of people in human history. Just enormous. And to be anxious about that is very reasonable."
He said it was "really important" that fears were listened to, and resources put in place to address them.
Referendum fear 'valid'
Archbishop Welby also described fear about what would happen if the UK left the EU or remained as a "valid emotion" .
He adds: "I don't think there is one correct Christian view, one way or the other. You can't say 'God says you must vote this way or that way'."
Work and Pensions Secretary Iain Duncan Smith, who is one of five cabinet ministers campaigning for the UK to leave the EU, welcomed the archbishop's comments on migration
He said: "These are rational comments from the archbishop - they're to be welcomed - but you wonder just how late they've come from various people in institutions, so I congratulate him.
"If you think back, for far too many years what's happened is that in a sense the elites have all said 'It's terrible to talk about immigration and if you do you're racist', so they've shut down the debate for many, many years."
Mr Duncan Smith said this was leading the debate about immigration to be "pushed to the margins, which is what you're seeing in Europe, and then political parties and people with very poor intentions and rather nasty motives then start to take this issue and that's where you lose control".
He added there should be a "system that controls migration", which he said would make it easier for people to assimilate into the UK and "bring talents and skills that are necessary to that community".
The prime minister's official spokeswoman said Mr Cameron agreed with Archbishop Welby's comments about migration, saying it was a "legitimate concern".
She added that the PM had taken a "range of measures to reduce migration from outside the EU" and had made it "a priory in the renegotiation that we have just had with our membership of the EU, where he has secured reforms to address this issue".
Meanwhile, former prime minister Tony Blair called for pro-EU campaigners to make their case with "passion, vigour and determination".
In an interview with BBC Radio 4's Today programme, he said the UK's destiny was to "lead in Europe" and warned leaving the EU would cause economic instability and "damage fundamentally".
In response, the Vote Leave campaign group said the UK would hand over "more money and more power to Brussels in return for some hollow promises" if it remained the EU.