"I was disappointed and surprised," said the 53-year-old. "From my perspective it didn't represent the working relationship I thought we had."
Clegg has spent almost four years as the public face of the Ipswich board, working directly to owner Marcus Evans.
Simon Clegg factfile
Born 8 November, 1959 in Middlesex
British Olympic Association's chief executive for 12 years and Team GB's chef de mission for the 2008 Beijing Olympics
Awarded CBE in 2006 for his role in London's successful bid to host the 2012 Games
Became Ipswich chief executive in April 2009
He is a former army officer in the Paras
But Clegg has taken the brunt of fans' criticism for the club's inability to mount a serious promotion challenge in that time, despite making it his aim to bring Premier League football to Portman Road.
"In that regard, yes, it has been a failure," he conceded. "I wanted to secure success for the football club. I wanted to be part of that journey of taking us into the Premier League.
"The easy thing would be to say it's through lack of investment, but you can't argue that. Marcus has backed each of his managers since he has been at the club with considerable finance.
"I've signed 73 players since I joined the club four years ago. Fans wish to criticise someone - and as the public face of the club that is one of the responsibilities I have had to bear."
Evans, a businessman who has bankrolled Town for the past five years, has consistently maintained a low media profile.
Clegg has said his departure is, in part, designed to give Evans a more "hands-on" role, but does not expect the owner to be stepping into the spotlight.
"I think that is unlikely," he added. "He's not totally shared that with me yet. They're still working on that through."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.