Cambridge University rescinded a visiting fellowship offer to Dr Jordan Peterson because he was pictured with a man wearing an anti-Islam shirt, its vice-chancellor said.
A two-month offer to the psychology professor was removed last week.
Dr Peterson labelled the decision a "serious error of judgement".
But Stephen Toope said the university was prompted to rescind the offer due to the picture of a man wearing an "I'm a proud Islamophobe" shirt.
University of Toronto professor Dr Peterson, whose views on gender have been condemned by critics, was set to spend a period in autumn at Cambridge's Faculty of Divinity.
He claimed in a blog post that he planned to collaborate with the faculty on a lecture series on the Biblical book of Exodus.
Visiting fellows are not paid but can make use of the libraries, attend seminars and engage in research, according to the university.
Professor Toope said the offer of the fellowship was granted on an academic basis but, "as a consequence" of becoming aware of the image, the invitation was rescinded.
"The casual endorsement by association of this message was thought to be antithetical to the work of a Faculty that prides itself in the advancement of inter-faith understanding," he said.
Dr Peterson has previously said he refuses to comply with Canadian laws compelling him to use the gender-neutral pronoun of an individual's choice, and in 2016 released a video lecture series questioning political correctness.
He also took aim at a campus culture of "social justice warrior, left-wing radical political activists".
Last week, Dr Peterson said he had not heard about the rescinding of the offer from the faculty, which announced its decision on Twitter, and said it publicised the decision "in a manner that could hardly have been more narcissistic, self-congratulatory and devious".
The BBC has contacted his representatives over the vice-chancellor's claims.