Boris Johnson's remarks about Muslim women wearing the burka were "inflammatory and divisive", the UK's equalities watchdog says.
The former foreign secretary said people in burkas looked like "letter boxes" or "bank robbers".
The Equality and Human Rights Commission said his comments risked "vilifying Muslim women".
Dozens of complaints to the Tory Party are being investigated but Mr Johnson has stood by his remarks.
Some Conservatives are unhappy at the party's response, and former Tory leader Iain Duncan Smith told the BBC: "My general view is I don't think that an internal party system should be there to shut down MPs when they speak."
Comedian Rowan Atkinson expressed support for Mr Johnson in a letter to the Times newspaper, saying "no apology is required" for ridiculing religion.
But one of Mr Johnson's critics, Conservative Muslim Forum founder Lord Sheikh, told BBC Newsnight he received "vile" Islamophobic messages after calling for him to be thrown out of the Conservative Party.
The equalities commission is understood to have received a complaint from a member of the public about Mr Johnson's comments.
Its chief executive, Rebecca Hilsenrath, said: "The effectiveness of our democratic society depends on freedom of expression, and the expression of offensive and intolerant opinions is generally not unlawful.
"Boris Johnson's use of language in this instance, which risks dehumanising and vilifying Muslim women, is inflammatory and divisive. Political figures should lead by example, conducting debates in a responsible manner, and language such as this can inhibit legitimate dialogue."
Lord Sheikh said earlier this week that Mr Johnson should be kicked out of the party for his remarks.
And speaking to Newsnight on Thursday, he added that in his view Mr Johnson had "let the genie out of the bottle" by inflaming tensions with his comments.
Out of 75 emails he had since received about the issue, 15 were complimentary while the rest included "offensive language" and "horrible things, obscene things, about the Prophet Muhammad", Lord Sheikh told the BBC.
The complaints to the Conservative Party against Mr Johnson are being looked at by an internal party team, which will decide whether any should be examined by an independent panel. Ultimately complaints can end up with the party's board, which has powers that include suspension or expulsion.
A source close to Mr Johnson offered no comment on the move while a Tory spokesman said: "The code of conduct process is strictly confidential."
But Blackadder and Mr Bean creator Atkinson told the Times: "I do think that Boris Johnson's joke about wearers of the burka resembling letterboxes is a pretty good one.
"All jokes about religion cause offence, so it's pointless apologising for them."
The comedian and comedy writer has previously campaigned against legislation that made it an offence to incite religious hatred on the basis that it could stifle freedom of speech.
Ex-Tory Chief Whip Andrew Mitchell has also said Mr Johnson should not have to apologise.
He said Mr Johnson had "expressed himself in quite colourful language, but he hasn't committed any offence and we need to be very careful about our handling of this because we believe in free speech in this country".
The Conservative Party has been accused of not doing enough to tackle anti-Muslim prejudice in its ranks, despite an initiative to boost tolerance and diversity.
The Conservative Muslim Forum and the Muslim Council of Britain had already raised concerns before Mr Johnson's article.
One hundred Muslim women who wear the niqab or burka have signed a letter to Conservative Party chairman Brandon Lewis, calling on him to withdraw the Conservative whip from Mr Johnson - whose Uxbridge constituency was targeted by around 30 protesters calling for his resignation - and launch an independent inquiry into Islamophobia in the party.
"We are not forced to make these clothing choices, nor are we oppressed," the women wrote in their letter, which has been issued to the media by the Muslim Council of Britain.