Some Muslim prisoners convicted of terrorism, and gang leaders, described as "very dangerous men", have been accused of bullying in a prison.
They had tried to "pressurise" other prisoners and were an "adverse influence" at the high security jail, HMP Whitemoor in Cambridgeshire.
The report, by HM Chief Inspector of Prisons, said most inmates felt safe.
The National Offender Management Service (NOMS) said the prison managed "a diverse population" very well.
Inspectors also reported they were concerned staff used excessive force instead of trying to "de-escalate" situations.
The report identified the dangers of a gang culture at the prison which housed 454 adult men at the time of the inspection in January.
All were serving long or indeterminate sentences for very serious offences.
"There were some very dangerous men, some of whom tried to influence and pressurise other prisoners," the report said.
"In some cases this was gang-related, and included some Muslim prisoners convicted of terrorist offences who were an adverse influence on others."
The prison held a disproportionately large Muslim population who accounted for approximately 40% of inmates.
Chief inspector Nick Hardwick said: "Overall Whitemoor was a safe, respectful and purposeful prison.
"The prison was doing some good work to manage its very diverse population and to understand and address the concerns of black and minority ethnic and Muslim prisoners.
"However, this remained a major challenge that needed a consistent high level of attention," he said.
Michael Spurr, chief executive of NOMS, said: "Whitemoor manages very challenging and long term prisoners so it is pleasing that the chief inspector has recognised the safe and purposeful environment.
"Staff will now use the recommendations in the report to address the areas of improvement identified."