Comments on slavery by David Starkey have been criticised for being racist.
The TV historian and author told an online show hosted by the conservative commentator Darren Grimes that slavery was not genocide because of the survival of "so many damn blacks".
Writing on Twitter, former chancellor Sajid Javid said: "David Starkey's racist comments are a reminder of the appalling views that still exist."
Starkey has not yet responded to the BBC's request for comment.
Fitzwilliam College, part of Cambridge University, where Starkey holds an honorary fellowship, described his remarks as "indefensible".
"We support and promote freedom of speech in our academic community, but we have zero tolerance of racism," the college said in a statement on Thursday. "Dr David Starkey's recent comments on slavery are indefensible."
They added: "The matter of Dr Starkey's honorary fellowship will be considered by the Governing Body at its meeting next Wednesday."
The Mary Rose Trust said it was "appalled" by Starkey's comments, adding on Thursday evening they had accepted his resignation.
Starkey made the offensive remarks in an episode of Grimes's YouTube show Reasoned, entitled "Dr David Starkey: Black Lives Matter Aims To Delegitimate British History".
The show's host tweeted on Thursday: "I reject in the strongest possible terms what Dr Starkey said in that clip and so very wish I'd caught it at the time. I am still learning the ropes, I will be much more alert to challenging this kind of thing in future."
In it, Starkey said: "Slavery was not genocide otherwise there wouldn't be so many damn blacks in Africa or Britain would there? An awful lot of them survived."
He also claimed that the Black Lives Matter protests, following the death of George Floyd, had been characterised by "violence" and "victimhood".
He described cancel culture and the pulling down of statues as "deranged".
The academic went on to discuss the links between slavery and the British Empire.
Starkey said: "As for the idea that slavery is this kind of terrible disease that dare not speak its name, it only dare not speak its name, Darren, because we settled it nearly 200 years ago."
"We don't normally go on about the fact that Roman Catholics once upon a time didn't have the vote and weren't allowed to have their own churches because we had Catholic emancipation."
Starkey's comments were heavily criticised by several social media users.
Nicholas Guyatt, a lecturer at the University of Cambridge, tweeted: "Can't speak for my employer but as someone who teaches history at Cambridge I'm ashamed of our connections with David Starkey and urge both the University and Fitzwilliam College to cut all ties with him."
It's not the first time Starkey has been involved in a public race row.
In 2011, the BBC received nearly 700 complaints about Starkey's claim that "whites have become black", during a Newsnight discussion about riots in the UK.