Broadmoor Hospital has been told to make urgent changes after a report which highlighted the "overuse" of face-down restraints on patients.
The Care Quality Commission (CQC) rated the high-security hospital in Berkshire "inadequate" after an inspection.
Inspectors said low staff levels and low morale were affecting safety and the quality of services.
The hospital's trust said it had been rated "good" in many other areas but pledged to make improvements.
West London Mental Health NHS Trust was told it must improve its practices in relation to the use of restraints and seclusion, which was an issue across the trust.
Restraint was being used mostly on acute, forensic and high secure in-patient wards, including Broadmoor, the report found.
In 179 out of 432 cases, patients were restrained in the prone position, also known as face-down restraint, which the government considered banning in 2013.
Dr Paul Lelliott, the deputy chief inspector of hospitals and lead for mental health, said: "We were concerned at the apparent overuse of physical restraint.
"Staff must use restraint only as a last resort, and minimise the use of restraint in the prone position."
Three main areas of concern across the trust:
- "A substantial problem with staff recruitment and retention. There were too few staff to consistently guarantee safety and quality in the forensic services, high secure services and community based mental health teams."
- "The trust must improve its practices in relation to restrictive interventions such as the use of restraint and seclusion."
- "The trust had a problem with low morale and poor engagement with front-line staff in some of its services."
The trust, which covers numerous mental health wards and psychiatric intensive care units in and around London, was told it must ensure staffing levels are sufficient and morale is improved after inspectors said there were too few staff to guarantee a safe and quality service.
Across the trust, 25% of staff surveyed said they experienced physical violence from patients and relatives, compared to the national average of 18%.
In line with other high secure services, Broadmoor, based in Crowthorne, had withdrawn additional payment for bank staff and newly appointed staff, which the CQC believes has had an impact on staffing levels.
Overall the trust was rated as "requiring improvement" but was told it must address highlighted changes as a "matter of urgency".
Trust chief executive Steve Shrubb said: "The inspection was rigorous and the CQC's assessment of the trust is very fair.
"They recognised our staff's hard work and compassion, many of our services were found to be good, and all of our services were found to be caring and staff should feel rightly proud of this.
"However, we know that we have more work to do and need to deliver improvements if we are to deliver consistently high quality care to every single patient.
"I apologise to patients and their families if they have received poor care from us."