David Cameron has been accused of "lazy and misguided" policy-making after he said a poor grasp of English left Muslim women in the UK "more susceptible" to the lure of extremism.
Baroness Sayeeda Warsi - a former Conservative chairman - said he was guilty of "stereotyping communities".
The PM said he wanted to help women learn English to improve integration.
But he also suggested failing to learn English could affect people on spousal visas who wanted to settle in the UK.
The government says 22% of Muslim women living in England speak little or no English - a factor it argues is contributing to their isolation.
Mr Cameron has announced a £20m fund to provide English lessons in homes, schools and community facilities. The money will be targeted at "specific communities" identified by a review into segregation that is being conducted by Louise Casey, head of the government's "troubled families" unit.
Speaking to BBC Radio 4's Today programme, Mr Cameron said the push on language was "about building a more integrated, cohesive, one nation country where there's genuine opportunity for people".
He also said that while he accepted there was no "causal connection" between poor English and extremism, a better grasp would make communities "more resilient" to threats of radicalisation from so-called Islamic State - or Daesh.
"If you're not able to speak English, you're not able to integrate, you may find, therefore, that you have challenges understanding what your identity is and you could be more susceptible to the extremist message that comes from Daesh," he said.
Lady Warsi - who was the first female Muslim cabinet minister - welcomed the new money for language teaching, but told The World at One: "This lazy and misguided linking, and what I saw once again as stereotyping of British Muslim communities, I felt took away from what was a positive announcement."
She went on: "I think it is lazy and sloppy when we start making policies based on stereotypes which do badly stigmatise communities."
New rules will mean that, from October, people coming to the UK on five-year spousal visas will have to take a test half way through that period to show they are making efforts to improve their English.
Asked what would happen to those who failed, Mr Cameron said: "They can't guarantee that they'll be able to stay."
He later expanded, saying that if they were not found to be making progress after two-and-a-half years there would be no guarantee they would be able to "go to the full stage and retain their visa".
The BBC's political correspondent Alex Forsyth said the government was absolutely not suggesting people could be deported if they failed to reach the required level, but that language skills would be one factor taken into account when deciding whether to extend a person's right to remain.
Lady Warsi said "threatening" women - even those who have children in the UK - with being "sent back" was "a very unusual way of empowering and emboldening women".
Mohammed Shafiq, chief executive of the Ramadhan Foundation, accused Mr Cameron of "using British Muslims as a political football to score cheap points to appear tough".
"There are three million Muslims in this country and the prime minister chooses to focus on a very small minority of extremists when clearly the majority of British Muslims reject extremism."
Women arriving in the UK under a spousal visa are currently expected to have English skills at the internationally-recognised A1 beginner level - roughly equivalent to a native-born child starting primary school.
Under the new rules, the women would be expected to have reached the A2 - elementary - level after two-and-a-half years, and B1 - intermediate - after five years.
Shuja Shafi, secretary general of the Muslim Council of Britain, welcomed the plans for language tuition but said: "The prime minister's aim to have English more widely spoken and for better integration falls at the first hurdle if he is to link it to security and single out Muslim women to illustrate his point."
Shadow home secretary Andy Burnham said Mr Cameron's "clumsy and simplistic" approach was "unfairly stigmatising a whole community".
"There is a real danger that it could end up driving further radicalisation, rather than tackling it."