Boris Johnson has stood by his remarks about the burka after the Conservative Party chairman told him to apologise.
The former foreign secretary has been criticised for saying Muslim women wearing burkas "look like letter boxes" and comparing them to "bank robbers".
PM Theresa May has backed calls for Mr Johnson to apologise, saying the remarks have "clearly caused offence".
But a source close to Mr Johnson said he "won't be apologising", adding it was "ridiculous" to attack his views.
"We must not fall into the trap of shutting down the debate on difficult issues," the source added.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
Mr Johnson's remarks, in a Daily Telegraph article, have provoked criticism from Muslim groups, some Tory MPs and opposition parties.
In the article, Mr Johnson said full-face veils should not be banned but that it was "ridiculous" people chose to wear them.
However, the founder of the Conservative Muslim Forum said the remarks would harm community relations.
Conservative Party chairman Brandon Lewis said he agreed with Foreign Office Minister Alistair Burt that there had been degree of offence in Mr Johnson's comments, and called on the former London mayor to apologise.
By BBC political correspondent Susana Mendonca
The criticism pitted at Boris Johnson has largely been around the words that he used - "letter boxes" and "bank robbers" to describe Muslim women wearing face veils.
But Mr Johnson hasn't directly responded to the accusation that it is his language which caused the offence.
Instead, the statement that has come from a source close to him suggests Mr Johnson wants to turn this into a debate about whether or not women should wear the burka at all.
Keeping a high profile is important to Mr Johnson.
He is seen by some in the Conservative Party as a contender for leadership - particularly since his resignation in which he criticised the prime minister's plan for Brexit.
But he risks being seen as weak if he is forced into an apology by the party chairman.
So his refusal to apologise, and an attempt to now put himself on the side of "liberal values", could be an example of Mr Johnson trying to save face while maintaining his profile.
Former Conservative chairwoman Baroness Warsi, the first Muslim woman to sit in a British cabinet, welcomed Mr Lewis's intervention and called for disciplinary action against Mr Johnson if he did not apologise.
She described the remarks as "offensive and deliberately provocative, but very clever politics".
A debate about the burka should be had "in a serious way", she said, rather than "trying to get airtime and attention on an issue which he knows will resonate with a certain part of the Tory Party".
Shazia Awan-Scully, a Muslim who ran as a Conservative candidate in 2010, said she did not agree with wearing the burka but it was up to women to make up their own minds.
She told BBC Radio 4's PM programme: "Some women choose to wear it, some women are conditioned to wear it, but it's certainly out of the question for me to tell a woman what to wear."
She added that the language used by Mr Johnson showed he was "pandering to this Islamophobic attitude bubbling away in the Conservative Party", which she said she had experienced herself.
Some MPs oppose the banning of the burka, but want to banish grid girls from existence. You cannot expect a society that celebrates gay pride and embraces gay marriage to live harmoniously when condoning the suppression of women forced to cover up, segregate and become invisible— Nadine Dorries (@NadineDorries) August 7, 2018
Conservative backbench MP Andrew Bridgen said Mr Johnson had raised an important subject in a "light-hearted way".
The reaction "says a lot about internal Conservative Party politics", he told the BBC.
Dr Qanta Ahmed, a British-American Muslim doctor who supports banning face veils, said Mr Johnson had been "tactless".
The 50-year-old said wearing a full-face veil was a "new phenomenon" and she didn't consider it a religious right, adding that she would be "very opposed" to it becoming normalised in British society.
What is Islamophobia?
By BBC home affairs correspondent Dominic Casciani
Last year, Zaynab Hussein, a mother from Leicester, was left fighting for her life after a man she didn't know smashed into her with his car - and then ran over her again. She was attacked because she was a Muslim.
Tell Mama, the national organisation that collects anti-Muslim attack statistics, says that the majority of street victims of such abuse and violence are women, for the same reason that Mrs Hussein was singled out: some Muslim women are easily identifiable by their mode of dress - and therefore easy to target.
Seven years ago Baroness Warsi said prejudice against Muslims had passed the "dinner table test".
And Mr Johnson's critics regard his "letter box" and "bank robber" comments as part of the problem the peer defined: normalising prejudice and dehumanising women, rather than calmly debating the complexities of the veil in an open society.
Since Baroness Warsi's warning, there has been the launch of a cross-departmental working group to tackle anti-Muslim hatred.
But it has been criticised as toothless, not least because the government can't agree a definition for Islamophobia.
What Boris Johnson said
In his Daily Telegraph column, Mr Johnson - who last month quit the government in protest at Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West, and could lead to "a general crackdown on any public symbols of religious affiliation".