Private security firm G4S has lost its contract to run Wolds prison in East Yorkshire and failed to win a number of other private jail contracts.
A report by HM Inspectorate of Prisons in August said HMP Wolds had "clear weaknesses", with poor behaviour and high levels of drug use among inmates.
G4S said it was "disappointed".
The firm was heavily criticised for its handling of the Olympic Games security contract in the summer, but ministers said the jail decision was unrelated.
Two G4S directors resigned in September after a review found the firm had failed to take account of the unique and complex nature of its contract to supply security for the London Games.
Armed forces personnel had to be drafted in when G4S failed to recruit enough staff.
Meanwhile, it has emerged that on Wednesday four prison officers were injured in what is thought to have been an attack with glass by an inmate at Birmingham Prison, which was taken over by G4S last year.
The officers were treated in hospital and an investigation is under way.
Prisons Minister Jeremy Wright told BBC News that decisions against G4S had nothing to do with what happened before the Olympics.
He said: "We've assessed all of these bids on their merits and not on the basis of who is making the bid.
"We have been looking at how to deliver a better service across 131 prisons and have a better deal for the taxpayer at the same time."
BBC home affairs correspondent Danny Shaw said Ministry of Justice sources had indicated that G4S's bids were not "compelling enough".
'Package of reforms'
An MoJ spokeswoman said of the Wolds bid: "Our evaluation has concluded that it would be more beneficial to allow the current contract to expire and the prison to revert to public sector management."
Wolds will return to the public sector when G4S's contract runs out in July.
G4S currently runs six prisons: Altcourse, in Liverpool; Parc in Bridgend; Rye Hill, near Rugby; Birmingham Prison; Oakwood, near Wolverhampton; and Wolds.
G4S said: "As the leading private provider of prison management in the UK, we have 20 years of experience of running prisons for the Ministry of Justice.
"Our performance across all six prisons we run has been to a high standard with every aspect of performance either meeting or exceeding the key performance indicators applied by the MoJ.
"We look forward to discussing the contract award decision with the MoJ within the next few days to determine why we were unsuccessful."
Three companies - Sodexo, Serco and a MTC/Amey consortium - remain in the bidding process for HMP Northumberland - an amalgamation of Castington and Acklington prisons - and Moorland, Hatfield and Lindholme prisons, in South Yorkshire.
Durham jail, Coldingley prison in Surrey, and Onley in Warwickshire will remain in the public sector.
The Ministry of Justice said the prisons competition process had produced a "compelling package of reforms" which it hoped would save £450m over the next six years.
Justice Secretary Chris Grayling said: "The cost of running our prisons is too high and must be reduced. We can do this by being more innovative and efficient, and without compromising public safety."
Frances Crook, chief executive of the Howard League for Penal Reform, said the government's decision to hand over prisons to the private sector had been "a mistake of Olympic proportions".
She said: "The government will seek to deflect criticism of its prison privatisation programme by excluding G4S from the next stage of the bidding process, but the principle of awarding lucrative contracts to private companies running prisons on the cheap remains unchallenged."
The general secretary of the Public and Commercial Services union, Mark Serwotka, called for an independent review of the private prison system.
He said: "The privatisation of our prison service ought to be a national scandal and that this has happened without any public debate is shameful.
"It is morally reprehensible that companies are profiting from locking people up and we urgently need an independent review to look at the impact on our communities, staff and prisoners."
Andrew Haldenby, director of the independent think tank Reform, said the decision to return some prisons to the public sector was a "dramatic U-turn which calls into question the government's whole approach to improving public services.
"Competition has improved the prisons environment for both prisoners and staff. It has worked successfully for two decades under successive governments."