Boris Johnson has threatened to privatise the Passport Office if it does not start delivering better value for money, a source has said.
During a cabinet meeting, the PM is understood to have told ministers the high cost of services was affecting families ahead of the summer holidays.
Mr Johnson is expected to call its bosses for urgent talks next week.
If they cannot promise better value, No 10 will see if the "private sector can do it in their place".
A source also said the PM made the same threat about the DVLA "if it does not get its act together", with Mr Johnson said to be concerned about how delays in issuing licences had impacted hauliers and supply chains.
The Passport Office - which reports directly to the Home Office - is dealing with a large backlog as the number of new applications rocketed after the pandemic and Brexit.
A government source said many families were being forced for pay for premium services because of delays in the system, and travellers have reported needing to cancel holidays.
They claimed the issue "goes to the heart of the cost of living" and "basic competence" of arms-length government bodies.
But the Public and Commercial Services Union (PCS) said the threat of privatisation was "absurd" when "it is clear that the current problems are mainly down to increased casualisation of the workforce".
The union said a lack of staff and poor management were among the causes of the backlog in processing passport applications.
Home Office Minister Kevin Foster told MPs on Monday the service had processed one million applications last month, compared to seven million for the whole of last year.
He said the government had no intention of letting the current processing time slide further, but urged people to apply to renew outdated passports "as soon as possible".
A HM Passport Office spokesperson said the "vast majority of all passports applications are being dealt with well within 10 weeks".
"To deal with this demand, we have increased staff numbers by 500 since April 2021," the spokesperson said. "This has helped us to handle more applications than ever before."
The rise in passport applications is thought to be down to people putting off renewing while they could not travel during the pandemic, with the Passport Office estimating five million applications were delayed in 2020 and 2021.
It has also been linked to a change in the rules for travelling to some EU countries post-Brexit, because passports must be no more than 10 years old from the point of issue.
Earlier this month, the director general of the Passport Office, Abi Tierney, advised people to allow up to 10 weeks when applying for their passport, adding: "Our dedicated staff are working tirelessly to ensure that passport applications are processed as quickly as possible."
But the SNP's Stuart McDonald called it a "predictable mess", telling the Commons: "All our constituents are having to cancel holidays, miss funerals, rearrange visits, with even a new 10-week target routinely being failed."
Labour MP Stephanie Peacock also told MPs about a mother in her constituency who had submitted her daughter's passport application in January - five months in advance of their holiday next week - but has yet to receive one.
She asked: "What is the minister doing to address the unacceptable delays in passport applications?"
Home Secretary Priti Patel said she would examine the case, adding: "That's a very, very unusual delay, there must be a problem."
The boss of Heathrow Airport, John Holland-Kaye, called the delays "disappointing".
He told the Press Association: "Normally the Passport Office is very slick in processing passports. If it's not resolved, it is going to mean that people can't get away.
"After two years of lockdown, we need to make sure people can have a good holiday because many of them have vouchers that they've been saving up, they've got trips of a lifetime that they've put on hold, and we don't know when things might get closed down again.
"Making the most of the summer is vital."