Many families in England could be forced out of their council homes and be unable to afford alternative housing in the same area, new research says.
Subsidised rents for households earning more than £30,000, or £40,000 in London, will be scrapped in April 2017.
Social housing tenants will be asked to pay rent at or near market rates.
Ministers say it "better reflects people's ability to pay" - but Labour and the Local Government Association say it would hit hard-working families.
The department for communities and local government argues that is "not fair that hard-working people are subsidising the lifestyles of those on higher than average incomes, to the tune of £3,500 per year".
Housing Minister Brandon Lewis said people who could afford to pay more in rent "should do so" - but it would be graded, so that those on lower incomes would not have to pay full market rates.
"We have always said we would consider carefully how much more people pay as they move through the pay scales, once they get over £30,000," he told BBC Radio 5 live's Pienaar's Politics.
He said the government would "listen to what comes in before we make a final decision in the period ahead of us".
According to the LGA-commissioned study, almost 60,000 households in England will not be able to afford to pay rents at the market rate or take advantage of the right to buy.
Councillor Peter Box, housing spokesman at the LGA, which represents hundreds of local councils in England, said: "Many social housing tenants across the country will be unable to afford market rents or take up the offer to buy their council home under this policy.
"A couple with three children, earning £15,000 each a year, cannot be defined as high income. Pay to stay needs to be voluntary for councils - as it will be for housing associations.
"This flexibility is essential to allow us to protect social housing tenants and avoid the unintended consequence of hard-working families being penalised, people being disincentivised to work and earn more and key workers, such as nurses, teachers or social workers, having to move out of their local area."
The LGA figures, compiled by the estate agent Savills, show that almost 215,000 social housing tenants will be affected by the "pay to stay" policy.
The policy is expected to save £245m a year by 2019-20, ending a situation where higher-income social tenants benefit from taxpayer-funded subsidies of up to £3,500 per year.
Former head of the civil service, Lord Kerslake, now a crossbench peer, will attempt to amend the housing bill, which is being scrutinised next week in the House of Lords, to place the scheme at the discretion of local authorities.
The peer, who is now chairman of the Peabody Trust housing association, has warned that the proposals were part of a package that could "threaten the future of social housing".
He told The Observer: "When this was originally discussed in the coalition government, it was intended to deal with the very small number of high earners on over £60,000.
"The current proposals will affect a lot more households with earnings of half that."
Labour's shadow housing minister John Healey said: "This new research backs Labour's concerns that this Tory 'tenant tax' could force thousands of working families from their homes.
"As the Housing and Planning Bill is debated in the House of Lords over the coming weeks, ministers should take a hard look at the effect their plans will have and use this opportunity to think again."