Justice Secretary Robert Buckland is facing legal action over his handling of the coronavirus outbreak in prisons across England and Wales.
Two charities claim his plans to tackle the spread have been "too slow" to make an impact, making them "unlawful".
The government says up to 4,000 prisoners could be let out up to two months early to create more space - but on Tuesday, only 18 had been released.
The government said it had robust plans in place to protect prisoners.
But Labour's shadow justice secretary David Lammy said: "Time is of the essence to prevent our prisons from becoming Covid-19 hotspots."
By Wednesday, 232 prisoners had tested positive for Covid-19 across 60 jails.
A total of 96 staff had also been infected so far, with coronavirus detected in over half of all prisons across the two countries.
The Prison Governors' Association has called for the prison population to be reduced by 15,000.
The Howard League for Penal Reform and the Prison Reform Trust issued a formal legal letter - the first step in judicial review proceedings - on Friday.
The charities said it came after "weeks of urgent correspondence" between them and the government, calling for faster action.
'Rise to this challenge'
Chief executive of the Howard League, Frances Crook, said Mr Buckland had "accepted publicly that the number of people in prison must be reduced significantly in order to save lives".
But she said it could not be achieved by the measures the government has in place.
"The rate of infection is accelerating, and the window of opportunity to protect people is vanishing," she added. "Ministers must rise to this challenge and act immediately to avert a public health catastrophe."
Peter Dawson, director of the Prison Reform Trust, said the government's actions when it came to prison had been "too little, too late".
He added: "The scientific and operational advice couldn't be clearer - if ministers are serious about following it, they must go much further, and do it now."
The Ministry of Justice said it would respond to the letter in due course.
A spokesman said: "We have robust and flexible plans in place keep prisoners, staff and the wider public safe based on the latest advice from Public Health England.
"As part of the national plan to protect the NHS and save lives, we have already announced up to 4,000 risk-assessed prisoners who are within two months of their release date will be temporarily released from jail, along with pregnant women."
But Labour's Mr Lammy said there were still a number of issues, including inadequate PPE for staff and sparse testing of inmates.
He added: "We hope the secretary of state for the Justice takes this warning shot by these leading criminal justice[organisations] very seriously.
"Urgent action is needed to prevent a public health disaster in the justice system."