Ministers have defended the choice of Sir Roger Scruton to head a new housing body, amid calls for him to quit.
The conservative academic was recently named as chairman of a new public body to encourage better designed homes.
But Labour said past "hateful" comments about homosexuality, which he once said was "not normal", and Islamophobia, which he called a "propaganda word", raised questions about his suitability.
Communities Secretary James Brokenshire said his views had been misrepresented.
The philosopher's character had been "smeared", Mr Brokenshire told MPs, saying he was "uniquely qualified" for the role because he was a world-leading authority on aesthetics.
One of the UK's leading public intellectuals and the author of more than 50 books, Sir Roger was named as chairman of the Building Better, Building Beautiful commission earlier this month.
The unpaid appointment has divided opinion, with Labour and Lib Dem MPs saying he is unsuitable because of remarks he has made in the past on social, religious and moral issues.
In an article for the Daily Telegraph in 2007, Sir Roger wrote: "Every now and then, however, we wake up to the fact that, although homosexuality has been normalised, it is not normal."
In an interview for online magazine Spiked in 2016, he said: "Lesbianism is usually an attempt by a woman to find that committed love that she can't get from men any more."
In a 2005 lecture in the United States, he said date rape was not a crime and what "supposed" victims were actually saying was: "'The whole thing went too quickly,' you know, 'I was not prepared,' and so consent is withdrawn as it were in retrospect."
Speaking in Hungary in 2016 on the future of Europe and the nation state, he made remarks about the financier George Soros which one Labour MP has said amounted to "peddling anti-Semitic conspiracy theories".
"Many of the Budapest intelligentsia are Jewish and form part of the extensive networks around the Soros empire," Sir Roger said.
He added: "As the world knows, indigenous anti-Semitism still plays a part in Hungarian society and politics and presents an obstacle to the emergence of a shared national loyalty among ethnic Hungarians and Jews."
Raising the appointment in the House of Commons, shadow communities secretary Andrew Gwynne suggested Sir Roger had made a number of other "unacceptable" comments, including "speaking favourably" of far-right movements such as the National Front.
"If we are going to have a society which welcomes free speech, we should hold people to account for what they use that privilege to say," he said.
His Labour colleague Clive Efford said what was at stake was not defending freedom of speech but whether someone who had "extreme views" was suitable for a high-profile public role.
But a succession of Conservative MPs offered their support to Sir Roger, saying the selective quotes cited by his critics were not a proper reflection of his arguments.
Jacob Rees-Mogg said as a philosopher his role was to "advance debate and discussion" on difficult issues, while Bob Seely said the government should not allow itself "to be bullied by the thought police rent-a-mob".
Mr Brokenshire said "due diligence" had been carried out before the appointment and ministers did not have to agree with all Sir Roger's "strong and controversial views", some of which he said he had changed his mind on, to recognise the contribution he could make.
"It saddens me that someone who has done so much to champion freedom of speech, expression and thought should be subject to such misinformed, ill judged and personal attacks we have seen over the last few days."
Sir Roger has said he is "offended and hurt" by suggestions that he is any way anti-Semitic or Islamophobic, insisting "nothing could be further than the truth".
In a statement on his blog last week, he said: "If people actually read my comments regarding the interplay between George Soros and Hungary, they will realise they are not in any way anti-Semitic, indeed quite the opposite.
"My statements on Islamic states points only to the failure of these states, which is a fact.
"My views on Islam are well known and can be found in my book The West And The Rest."