Boris Johnson must "lead or step aside", a senior Tory and former minister has told the BBC.
Tobias Ellwood, who chairs the defence select committee, said "we need leadership" following reports of parties being held in Downing Street while Covid restrictions were in place.
It comes as hundreds of angry constituents contact their MPs.
The government has urged people to reserve judgement until senior civil servant Sue Gray's inquiry is finished.
But Andrew Bridgen, Conservative MP for North West Leicestershire, said he does not need to wait for Ms Gray's investigation to know that the PM "has lost the moral authority to lead the country".
Mr Bridgen became the fifth Conservative MP to publicly declare they had written to the chairman of the 1922 Committee - which organises Tory leadership contests - to say they have no confidence in the prime minister. Fifty-four Conservative MPs have to write a letter to trigger a vote.
The latest Conservative MP to call for the PM to stand down is Tim Loughton, a former children's minister, who said Mr Johnson's position was now "untenable".
"Obfuscation, prevarication and evasion have been the order of the day when clarity, honesty and contrition was what was needed and what the British people deserve," Mr Loughton said in a Facebook post.
Mr Loughton added he had some "lively conversations" with constituents on Saturday.
Ahead of Mr Johnson's apology on Wednesday, Mr Ellwood, MP for Bournemouth East and one-time defence minister, said Mr Johnson needed to "show some contrition" and get a grip of the situation - or he would be "out of office".
Another former minister told the BBC: "Johnson is toast... if you were the chief whip looking at him you'd say he's not fit to do any other jobs in government, you wouldn't make him a junior minister, he doesn't work hard enough."
And a senior Tory MP said "there is a lot of scepticism around that there is anyone ready to take the reins. That buys Boris time. But he shouldn't confuse that with another chance."
Some Tory MPs said their inboxes have filled up after No 10 apologised to the Queen for two staff parties the night before the Duke of Edinburgh's funeral.
Mr Johnson was not at either of those parties last April - but he faces questions over alleged Covid rule-breaking at No 10.
One senior backbench Conservative MP told the BBC they received more than 200 angry emails about the parties, with only five messages in support of the prime minister.
"Many colleagues now believe Boris won't be leader at next general election... for many of us this feels terminal," the MP said.
A Midlands Tory MP, who won his seat in a former Labour constituency in the 2019 election, said: "The inbox is bad, really bad."
The gatherings which Downing Street has apologised to Buckingham Palace for took place on 16 April 2021.
The two parties last April involved about 30 people in total, and are reported to have converged at some point in the Downing Street garden, where they continued past midnight.
Staff were reportedly sent to a nearby shop with a suitcase, which was brought back "filled with bottles of wine".
At the time, England was under "step two" restrictions that stipulated people could not socialise indoors, except with those from their household or support bubble. People could socialise outdoors in groups of up to six people or two households.
The behaviour of No 10 staff has been contrasted with pictures of the Queen sitting alone at the Duke of Edinburgh's funeral because of Covid restrictions.
Mr Johnson's spokesman said it was "deeply regrettable that this took place at a time of national mourning".
Ms Gray is carrying out the investigation into alleged Covid rule breaking in Downing Street and government departments.
Labour leader Sir Keir Starmer said it was now in the "national interest" for the prime minister to resign, saying he had lost his "moral authority".
The Liberal Democrats and the SNP also called for Mr Johnson to resign after he admitted attending a drinks party in the Downing Street garden during lockdown on 20 May 2020. The prime minister said he "believed implicitly that this was a work event".
Speaking to BBC's Politics North, pensions minister Guy Opperman said Mr Johnson's behaviour had been "unacceptable".
The Conservative MP said he felt particularly emotional about the party held in the Downing Street garden on 20 May because around that time Covid restrictions had prevented him from being with his wife and twins when they were ill in hospital. The two boys subsequently died shortly after birth.
Mr Opperman said the prime minister should continue in post while the investigation into government gatherings was completed. But he said Mr Johnson needed to "change his ways" and run Downing Street "in a very, very different way".
It comes as the former head of the government's Covid taskforce has apologised for having leaving drinks in the Cabinet Office when she left the civil service on 17 December 2020, when strict restrictions were in place in England.
Kate Josephs, who is now the chief executive of Sheffield City Council, said she was "truly sorry" for "the anger that people will feel".
More than 40 people were on the guest list, among them senior Downing Street staff and civil servants - and the event appears to have been held in a single room, BBC Newsnight has learned.
No 10 said the inquiry would establish the facts around the nature of the gatherings, including attendance, the setting and purpose - and make them public "in due course".