Toby Young says he regrets "politically incorrect" remarks he made in the past and should be judged on his actions in promoting social mobility in education.
The right wing journalist has faced a backlash on social media after it was announced he had been appointed to the board of a new university regulator.
Left wing critics have attacked his suitability for the role.
Boris Johnson has defended him but Tory MP Margot James said he was mistaken to appear to "belittle sexist comments".
Following the furore over his appointment, Mr Young said he had deleted 40,000 tweets sent before 2017 because a "handful of things" dating back to 2009 were being "wrongly misconstrued" as evidence he was opposed to gender equality and LGBT rights.
However, Labour said this was not sufficient action and his appointment should be revoked.
"If Theresa May doesn't rethink this appointment, the message she is sending to students is that under the Tories, misogyny and homophobia will not just be tolerated, but rewarded," said Dawn Butler, shadow minister for women and equalities.
Students and academics have, meanwhile, taken to Twitter to also call for his removal, highlighting disparaging and crude remarks he has made about women's breasts in social media posts and a 1998 article for Gear magazine on "being a lesbian for the night".
An online petition calling for Mr Young to be sacked from his post because of past remarks has attracted more than 28,000 signatures.
A lecturers' union has complained that Mr Young - who helped to found the West London Free School - has no relevant experience and that he had only been chosen because he is a "Tory cheerleader" and passionate supporter of the party's education policies.
But in a staunch defence on his blog his suitability to join the board of the Office for Students, Mr Young said he had championed meritocracy and rigour in education for decades and while he accepted that some of his past remarks had been in bad taste he rejected accusations of misogyny.
Some of those things have been sophomoric and silly – and I regret those – but some have been deliberately misinterpreted to try and paint me as a caricature of a heartless Tory toff. 23/— Toby Young (@toadmeister) January 3, 2018
Toby Young's "caustic wit" includes leering at politicians' and celebrities' breasts, talking about "hardcore dykes", calling a celebrity "as queer as a coot", and joking about gay celebrities wanting to "bum" straight men. Our Foreign Secretary thinks it's hilarious, though. https://t.co/Mw7I5eZvSM— Owen Jones🌹 (@OwenJones84) January 3, 2018
In a 2012 Spectator column, Mr Young hit out at the "inclusivity" agenda in schools, suggesting it meant "wheelchair ramps, the complete works of Alice Walker in the school library (though no Mark Twain) and a Special Educational Needs Department that can cope with everything from Dyslexia to Munchausen Syndrome by Proxy."
He told the BBC that the article had been misunderstood and he had since made clear that he was not disparaging inclusion in the "narrow sense of providing equal access to mainstream schools for people with disabilities" but criticising the notion of "a dumbed down, one-size-fits-all curriculum".
Given the volume of tweets he had sent over the years, he said it was "not altogether surprising" that anyone trawling through them had found some "sophomoric" remarks.
"I regret these, obviously. I hope people will judge me on my actions, not on silly things I've tweeted or written in my 30-year career as a journalist," he said.
Defending his qualifications, he said he had been "a passionate advocate" of widening participation and increasing social mobility since the mid-1980s when, as a state school boy at Oxford, he first started visiting sixth forms in deprived parts of the country to try to persuade students to apply to elite universities.
Business minister Margot James said that while she still believed Mr Young was a worthy candidate for the job she questioned how he had responded to criticism of his conduct on social media.
Toby Young @toadmeister is worthy of his appointment to the OFS but it is a mistake for him to belittle sexist comments by labelling them “politically incorrect” a term frequently used to dismiss unacceptable comments about, and behaviour towards, women and minorities— Margot James (@margot_james_mp) January 3, 2018
Boris Johnson - brother of Universities Minister Jo Johnson, who created the Office for Students to protect free speech on university campuses - said the "outcry" over Mr Young's appointment was "ridiculous".
The foreign secretary, who worked with Mr Young when he was editor of the Spectator, said the writer's "caustic wit" was among the attributes that would make him an "ideal man for the job," adding that he had the "independence and rigour" to stand up for students' interests.
The Office for Students has been established to hold universities to account on issues like vice chancellors' pay and free speech on campus, with powers to fine universities which fail to meet the required standards.
Mr Young is one of 15 board members who were selected following an open appointment process. Others on the board include lawyer Simon Levine, Boots executive Elizabeth Fagan and barrister and academic Carl Lygo.