Segregation and social exclusion are at "worrying levels" and are fuelling inequality in some areas of Britain, a report has found.
Women in some communities are denied "even their basic rights as British residents", the Casey Review said.
Dame Louise Casey accused public bodies of ignoring or condoning divisive or harmful religious practices for fear of being called racist.
Communities Secretary Sajid Javid said he would study the findings "closely".
Dame Louise's review into the integration of minorities was commissioned by former Prime Minister David Cameron as part of the government's efforts to tackle extremism.
Among her recommendations were that immigrants could take "an oath of integration with British values and society" and schoolchildren be taught British values.
Her review said there was a sense that people from different backgrounds got on well together at a general level, but community cohesion "did not feel universally strong across the country".
She found "high levels of social and economic isolation in some places, and cultural and religious practices in communities that are not only holding some of our citizens back, but run contrary to British values and sometimes our laws".
Her report highlighted the plight of women in some Muslim communities, who she said were less likely to speak English and more likely to be kept at home.
"Misogyny and patriarchy has to come to an end," Dame Louise said, adding that public institutions must not fear being branded racist or Islamophobic.
Faeeza Vaid, from the Muslim Women's Network, said migrant communities should not be blamed for failing to integrate.
"We also see segregated white communities," she told the BBC's Victoria Derbyshire programme.
"Integration is everyone's responsibility."
Halima Begum, a Muslim woman from London, said she was "worried" that the report singled out Muslim men.
"There is a problem with men and I see it," she told the BBC's Asian Network. "We need to have an open debate without placing too much blame on communities."
Muslim areas under spotlight
By BBC political correspondent Ross Hawkins
Afraid of being dubbed racist, afraid of losing support, afraid of challenging minority communities - that is Dame Louise Casey's view of Britain's decision-makers.
Criticising politicians and officials is the easy bit. The significance of this report is that it targets individual communities and faiths.
It contrasts with the language of people living "parallel lives" - a term used in an earlier report and designed to be neutral, placing no more blame on one community than another.
Dame Louise makes clear her outrage at what she calls "regressive practices" targeting women and girls. She accepts she is putting Muslim areas under the spotlight.
The question for Dame Louise - an official not immune to criticism herself - is how would she achieve change? What if an elected politician refused to take her oath? What if some people - of whatever community - simply prefer to live and educate their children separately?
Iqbal Bhana, a government adviser on anti-Muslim hate crime, rejected the report's claim that Britain was becoming more segregated.
"I don't think we are divided," he told BBC Radio 4's Today programme.
"If you go to places like Bradford or Leicester you see second and third-generation people moving out of some of these communities."
During her research, Dame Louise spoke to women in areas of Birmingham and Manchester that she said could not leave their homes without their husband's permission.
She blamed those in authority for "ducking the issue" of women's inequality out of a wish to respect different cultures.
"If [the women] were white and living in Surrey, we would all be up in arms about it," she said.
She said not talking about these issues would only "give ammunition to the extreme far-right and Islamic extremists", who are the people "who set out to divide us".
Labour MP Chuka Umunna, who chairs the all-party parliamentary group on social integration, agreed that division had bred extremism and was "sapping our communities of trust".
'Disadvantaged black men'
Immigration was another theme in the report, in which Dame Louise said some towns and cities were "struggling to adjust" to an influx of foreign nationals.
On a visit to Sheffield, she said, she had found some schools were coping with an increase in Eastern European children from 150 to nearly 2,500 over a five-year period.
The report also highlighted a "persistent disadvantage" facing black men in the workplace, as well as white working class children on free meals who underperformed at school.
The report made 12 recommendations, including:
- Projects to boost cohesion, such as local IT courses and sport activities for children
- Councils should regularly collect statistics on hate crime or deficiencies in English
- Schools should promote British values to help build integration, tolerance and citizenship
- A review of the "rights and obligations" of immigrants likely to settle in the UK
- Funding for school projects that encourage children of different backgrounds to mix
- More funding for local English language classes and a review of whether courses are reaching people who need them
- Councils should investigate whether their housing policies help or hinder integration
- New oath for public office-holders pledging "tolerance of those with different faiths and beliefs"
Mr Javid said Britain had "long been home to lots of different cultures and communities", but added that "all of us have to be part of one society".
While it was right to celebrate the "positive contribution" diverse groups make to British life, nobody should be excluded from it or left behind, he said.
"We need to take a serious look at the facts and must not shy away from the challenges we face.
"Dame Louise's report is a valuable contribution, and I will be studying her findings closely."
He said he would be reporting back in the New Year.