Such is the anger with the Speaker at senior levels of government, it has been suggested he could be blocked from getting a peerage when he retires.
Ministers are furious at what they see as John Bercow's "bias" during Commons debates on Brexit.
The move would break a tradition dating back 230 years, that former Commons Speakers are automatically offered a seat in the House of Lords.
A Cabinet source said: "It's a good job peerage nominations are in our gift."
They added: "I'm sure we'll be thinking carefully about which individuals we would choose to elevate to the House of Lords.
"I can't imagine we would look favourably on those who've cheated centuries of procedure."
By tradition, retiring Speakers have stood down as MPs at the same time triggering a by-election.
They are then awarded a peerage at the request of the Commons, in a motion asking the Queen to "confer some signal mark of Her Royal favour" upon them.
After a recommendation from Downing Street to Buckingham Palace, they then sit in the Upper House as a crossbencher, an independent.
This is what happened the last time a Speaker stood down.
Michael Martin was ousted ten years ago over his handling of the expenses scandal.
But, despite criticism at the time, he became Lord Martin of Springburn.
Lord Martin died last year.
John Bercow has faced criticism from MPs for months.
In October he told friends he intends to stand down as Speaker this summer, following a report which condemned a culture in Parliament in which abusive behaviour was "tolerated and covered up."
Mr Bercow has also faced allegations of bullying, which he has strenuously denied.
Earlier this month, plenty of people within the government were furious with the Buckingham MP, saying he broke precedent and ignored the advice of his officials in a heated row about parliamentary procedure over Brexit.
Critics within government saw this as proof that he was unable to be impartial over Brexit, and was determined to make life difficult for the government.
Two years ago, Mr Bercow revealed that he voted Remain in the EU referendum.
Asked about the prospect of being denied a peerage when he retires, a spokeswoman for the Speaker declined to comment.
A Downing Street spokesman said on Friday: "We don't recognise this story and we have quite enough to be getting on with at the moment."
Some Labour MPs have been tweeting their response:
It would be such a shame to lose John Bercow from the legislature given his robust defence of Parliament against abuses by the executive. If he can’t go to the Lords we’ll just have to keep on as Speaker for many, many years to come... https://t.co/Wa4MXUtKRK— Wes Streeting MP (@wesstreeting) January 17, 2019
It would be pathetic, petty and totally unreasonable for the Tories to punish John Bercow for doing his job, defending the rights and reputation of Parliament. They just don’t like anyone who stands up to them.— Owen Smith (@OwenSmith_MP) January 18, 2019
Brexit continues to divide parliamentarians in the wake of the crushing defeat of the prime minister's Brexit deal earlier this week, when it was rejected in the Commons by 230 votes.
Having narrowly survived a no-confidence vote on Wednesday, Theresa May has called on politicians from all sides "to put self-interest aside" and "work constructively together" to help her find a consensus for a new Brexit plan.
The PM will publish a new plan on Monday with a full debate and key vote scheduled for Tuesday 29 January.
So far she has held talks with senior figures from the SNP, the Lib Dems, the Green Party and Plaid Cymru - as well as members of the DUP and Tory Brexiteers.
But Labour leader Jeremy Corbyn remains adamant that he will not take part in the talks unless the threat of no-deal Brexit is ruled out.
In addition, Mr Corbyn has urged Labour MPs not to take part in any talks while a no-deal Brexit remains an option.
The prime minister has said it is "not within the government's power" to rule out a no-deal, and the "door remains open".