Inmates are living in cells without glass in the windows in conditions described as "squalid".
An inspection of Swinfen Hall prison near Lichfield, Staffordshire found almost all areas have deteriorated and the prison is "not safe enough".
The report also found inmates at the prison and young offenders institution were not having daily showers.
A spokesperson for the prison said the problems reflected "operational pressures".
The Chief Inspector of Prisons, Peter Clarke, said: "Basic standards to improve".
The report by HM Inspectorate of Prisons found:
- Most of the 180 cells had windows missing glass, exposing prisoners to the elements
- In the six months before the inspection, there were 79 assaults on prisoners and 24 on staff, as well as 60 fights
- Much of the accommodation, particularly on the older wings, was "squalid"
- Nearly half of inmates said they thought it was "easy" to get illegal drugs
- Many prisoners had to wait four days for a shower
The Chief Executive of the Howard League for Penal Reform charity, Frances Crook, said: "For the second time in as many weeks, we read of a prison where men are so frightened for their safety because of rising violence that they are refusing to come out of their cells.
"Many prisons are overcrowded, but Swinfen Hall is not, which makes this report particularly concerning."
Michael Spurr, Chief Executive of the National Offender Management Service, said more prison officers will be recruited to Swinfen Hall and an action plan is in place.
Inspectors noted the prison was working hard to address offending behaviour and work to resettle prisoners back into the community was "reasonably good".
In August 2015, the prison was put on lockdown. In December of the same year prison staff were taken to hospital after being attacked by inmates.
The unannounced inspection of the prison, which holds 600 male prisoners aged between 18 and 25, was carried out in October and November 2016.