Fitzwilliam College, at Cambridge University, has announced it has "accepted the resignation of historian David Starkey from his honorary fellowship with immediate effect".
The college said: "Our student and academic bodies are diverse and welcoming to all. We do not tolerate racism."
His comments on slavery were criticised on Thursday for being racist.
The TV historian has not yet responded to the BBC's request for comment.
Starkey told an online show hosted by conservative commentator Darren Grimes that slavery was not genocide, because of the survival of "so many damn blacks".
The master of Fitzwilliam contacted Starkey following his comments, and the college added: "Fitzwilliam prides itself in leading the way in Cambridge in opening access to higher education for underrepresented groups."
The statement went on to note that while the author "holds no teaching role" there, that "honorary fellows have the same responsibility as all members of our college to uphold our values".
Canterbury Christ Church University, meanwhile, has also announced it has "terminated David Starkey's position as visiting professor with immediate effect".
The university called the comments "completely unacceptable".
We have terminated David Starkey’s position as Visiting Professor with immediate effect. His comments are completely unacceptable and totally go against our University and community values.— CanterburyCCUni 🌈 (@CanterburyCCUni) July 3, 2020
Professor Rama Thirunamachandran, vice-chancellor of the university, apologised to staff and students who have been offended and upset by the "appalling" comments.
He said: "Widely reported comments by historian David Starkey during a recent online interview are, in our view, completely unacceptable and do not reflect the values of our university and community."
Lancaster University has also initiated a review of Starkey's honorary graduate status.
"His comments are abhorrent and contrary to our values," the institution tweeted.
Writing on Twitter, former chancellor Sajid Javid said: "David Starkey's racist comments are a reminder of the appalling views that still exist."
The Mary Rose Trust said it was "appalled" by Starkey's comments, adding on Thursday evening they had accepted his resignation.
Starkey made the offensive remarks in an episode of Darren Grimes's YouTube show Reasoned, entitled "Dr David Starkey: Black Lives Matter Aims To Delegitimate British History".
The show's host tweeted on Thursday: "I reject in the strongest possible terms what Dr Starkey said in that clip and so very wish I'd caught it at the time. I am still learning the ropes, I will be much more alert to challenging this kind of thing in future."
In it, Starkey said: "Slavery was not genocide otherwise there wouldn't be so many damn blacks in Africa or Britain would there? An awful lot of them survived."
He also claimed that the Black Lives Matter protests, following the death of George Floyd, had been characterised by "violence" and "victimhood".
He described cancel culture and the pulling down of statues as "deranged".
The academic went on to discuss the links between slavery and the British Empire.
Starkey said: "As for the idea that slavery is this kind of terrible disease that dare not speak its name, it only dare not speak its name, Darren, because we settled it nearly 200 years ago."
"We don't normally go on about the fact that Roman Catholics once upon a time didn't have the vote and weren't allowed to have their own churches because we had Catholic emancipation."
Starkey's comments were heavily criticised by several social media users.
Nicholas Guyatt, a lecturer at the University of Cambridge, tweeted: "Can't speak for my employer but as someone who teaches history at Cambridge I'm ashamed of our connections with David Starkey and urge both the university and Fitzwilliam College to cut all ties with him."
It's not the first time Starkey has been involved in a public race row.
In 2011, the BBC received nearly 700 complaints about Starkey's claim that "whites have become black", during a Newsnight discussion about riots in the UK.