The club went on to issue a statement saying he had been excluded because of "consistent criticism" of the board.
"I have given criticism so I accepted the ban," he told BBC Radio Bristol.
Chairman Nick Higgs said the incriminating remarks date back to when Spencer initially quit his role, and the ban had only recently been ratified because he had regularly started attending games again.
But Spencer argued: "Nobody had informed me of any ban.
"There was no reason given on the letter and I only heard through the AGM that it was historic."
Speaking on Saturday, Higgs said the club had offered Spencer a reprieve but he had declined clear-the-air talks.
"We have offered an olive branch for Kevin to come back," he said.
"We said there would be conditions attached but we were willing to talk to him about him returning, but he said in an email he did not wish to reconsider.
"I'm quite happy to go along with Kevin's wishes. He wants to draw a line under it and move on."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.