Nearly half of prisoners failed drugs tests at a prison that holds large numbers of gang members, a report said.
HMP Chelmsford has one of the worst rates of drug use in the UK, according to HM Inspectorate of Prisons, with organised gangs responsible for supplying illicit items.
There are "significant concerns about safety" and levels of violence are far too high, inspectors said.
Campaigners have blamed overcrowding and under-resourcing.
Peter Clarke, HM Chief Inspector of Prisons, said the increase in violence, suicides, availability of drugs and unacceptably poor living conditions led him to seriously consider invoking the Urgent Notification protocol, requiring the Justice Secretary to step in.
However, Mr Clarke said he had confidence in the prison's leadership to improve the situation, after a visit between 21 May and 7 June.
According to the report, 42.6% of prisoners failed drugs tests and much of the violence was down to the supply and use of illicit substances.
Inspectors said during the visit a member of the public was caught throwing illicit items into the prison from nearby trees and in one month illegal goods worth £15,000 were seized.
There have also been 17 suicides at the prison in the past eight years, including five since the previous inspection in 2016, one of which was since the latest visit.
Among the problems highlighted by the inspectors were that 40% of prisoners who did not attend activities were locked in their cells for up to 22 hours a day and there were shortages of items including mattresses and pillows.
The prison, which was built in the 1830s, held 700 men at the time of the inspection but has since reduced that by 50.
Prison reform group Howard League for Penal Reform said the prison was overcrowded, having been designed to hold 521 men.
Chief executive Frances Crook said: "The decision to move 50 prisoners out of Chelmsford is a tacit admission that overcrowding is a root cause of the prison's troubles."