Boris Johnson should be kicked out of the Conservative Party for his remarks about the burka, a Muslim peer says.
Conservative Muslim Forum founder Lord Sheikh told Newsnight demands from the PM for an apology were not enough.
He has called for Mr Johnson to have the whip removed - meaning the MP would no longer represent the Tory party.
Mr Johnson said Muslim women wearing burkas "look like letter boxes" and compared them to "bank robbers". A source said he stood by the comments.
Conservative peer Baroness Warsi - the first Muslim woman to sit in a British cabinet - said comments like Mr Johnson's could lead to a rise in hate crime in the UK.
Culture Secretary Jeremy Wright said there was no reason not to have a "robust conversation", but added: "We're not talking to our friends in the pub, we are public figures and we have an additional obligation to be careful."
Conservative Party chairman Brandon Lewis told BBC Radio 4's Woman's Hour it was right to have a debate on the issue but that Mr Johnson should say sorry for the "language that he used".
And a former Tory chairman, Lord Pickles, said Mr Johnson, who was foreign secretary until resigning last month over Brexit, risked "closing down" the debate with his "illiberal language".
He said that if someone made a complaint to the party, an independent panel would look into it, but it was "pretty much inconceivable" that the former foreign secretary would face disciplinary action.
A source close to Mr Johnson has said: "We must not fall into the trap of shutting down the debate on difficult issues.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
In his Daily Telegraph article, Mr Johnson said full-face veils should not be banned but that it was "absolutely ridiculous" women chose to "go around looking like letter boxes".
Theresa May has backed calls for an apology. The prime minister said the remarks had "clearly caused offence".
Lord Sheikh, a former adviser to David Cameron, said: "Take the whip from him. Why not? He's not a super human being, he's a member of the party. The party chairman, the prime minister has the right to take the whip.
"It's not out of order and that's the thing I'd like to see."
Mr Johnson, who is a former mayor of London and the current MP for Uxbridge and South Ruislip, has long been seen as a potential candidate for the leadership of the Conservative Party.
He fronted the successful Leave campaign in the 2016 EU referendum, and resigned as foreign secretary last month in protest at Theresa May's Brexit plans.
His comments have seen him accused of using the issue of Muslim women's veils as a "political football" to further his leadership ambitions.
Writing in the Guardian, Baroness Warsi said Mr Johnson's remarks were "indefensible" and "have no place in the modern Conservative Party".
She said although he was setting out a liberal position on the burka, he was doing it in an "alt-right" way, and using Muslim women as "political fodder… to stake out a leadership bid".
"Johnson's words… send out a message that Muslim women are fair game," she wrote.
"What starts as useful targets for 'colourful political language' and the odd bit of toxic campaigning ends up in attacks on our streets.
"As much as Johnson thinks he's being his usual clever self, he's helping to create an environment in which hate crime is more likely."
Lord Pickles told BBC Radio 4's Today programme: "I think the very sensible thing would be for him to apologise.
"If he is wanting to open up and defend liberal values, then the sensible thing is not to use illiberal language."
Speaking to BBC News later, he said the party was "far away" from removing the whip from Mr Johnson, but he said the MP was getting more "slack" than other party members might.
"If this was a councillor from Scunthorpe or Middlesbrough, I am not entirely sure he would have been given the amount of slack, and in truth he wouldn't have been given the amount of attention that Boris has got," he added.
But Mr Johnson's former adviser Munira Mirza said his views on the burka had been "entirely consistent" and other Conservative politicians had expressed the same view, without being called Islamophobic.
"The reality is there is a political fight here," she said. "People who frankly couldn't care less about the issues that Muslim women face are piling into Boris because Boris said it."
What is Islamophobia?
By BBC home affairs correspondent Dominic Casciani
Last year, Zaynab Hussein, a mother from Leicester, was left fighting for her life after a man she didn't know smashed into her with his car - and then ran over her again. She was attacked because she was a Muslim.
Tell Mama, the national organisation that collects anti-Muslim attack statistics, says that the majority of street victims of such abuse and violence are women, for the same reason that Mrs Hussein was singled out: some Muslim women are easily identifiable by their mode of dress - and therefore easy to target.
Seven years ago Baroness Warsi said prejudice against Muslims had passed the "dinner table test".
And Mr Johnson's critics regard his "letter box" and "bank robber" comments as part of the problem the peer defined: normalising prejudice and dehumanising women, rather than calmly debating the complexities of the veil in an open society.
Since Baroness Warsi's warning, there has been the launch of a cross-departmental working group to tackle anti-Muslim hate.
But it has been criticised as toothless, not least because the government can't agree a definition for Islamophobia.
What Boris Johnson said
In his Daily Telegraph column, Mr Johnson - who last month quit the government in protest at Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West, and could lead to "a general crackdown on any public symbols of religious affiliation".