MPs have rejected Labour calls for the government to revise aspects of its housing benefit reforms.
An opposition motion urging ministers to review plans for a 10% cut in benefit for those unable to find work within a year was defeated by 61 votes.
Several Lib Dem MPs expressed concerns about the plan during a lively Commons debate, while Labour accused the government of acting "recklessly".
But ministers said Labour was trying to "frighten" the public.
The debate on the proposed housing benefit reforms was tabled by Labour - who had hoped some unhappy Lib Dem MPs would back them in a vote calling on ministers to rethink some of the plans.
Lib Dem MP Bob Russell urged ministers to "think again" about the impact of benefit caps on families with children, while his colleague Jenny Willott expressed reservations about a proposed 10% cut in benefit for people on jobseeker's allowance for more than a year.
"If we are applying the test that these reforms are to support vulnerable people, help them into work and ensure they have a roof over their head whilst making sure work pays, then for me reducing housing benefits for job seekers after a year simply fails that test," she said.
But deputy Lib Dem leader Simon Hughes said there was "no evidence" the changes would lead to families being "forcibly moved" from their homes.
And the Labour motion was defeated by 319 votes to 258.
Shadow Work and Pensions Secretary Mr Alexander told MPs that press coverage suggested only "work-shy families in Mayfair mansions" would be affected by proposals for weekly housing benefit caps - but in fact it would be people, including pensioners and low paid workers, across Britain.
He said he supported the case for reforming housing benefit but said the current plans could ultimately "end up costing the taxpayer more".
Labour MP Dennis Skinner suggested the plans were similar to Dame Shirley Porter's plan to "shift people out of certain areas for political reasons".
Mr Duncan Smith pressed him to say whether he agreed with critics' claims that London would be "socially cleansed" were right - Mr Alexander said: "If these proposals pass unamended London will look very different in the years ahead."
But Mr Alexander said: "I have no objection in principle to a cap if introduced on a staged timetable. But we do have to raise the questions as to whether a national cap is the most appropriate plan."
He said he was more concerned about other plans - including changing the way rates of housing benefit are set so they are linked to the cheapest 30% of rental properties in an area and plans to cut housing benefit payments by 10% for unemployed claimants who have not found work within a year.
"There is a difference between having a duty to act and acting in such a precipitate and reckless fashion that it ultimately ends up costing the taxpayer more," Mr Alexander said
"This package of rushed, ill-considered and potentially devastating cuts have raised real concerns... in communities across the country."
Mr Alexander said housing benefit had risen by about 21% during the recession - due not to increased rates but to more people claiming. However Mr Duncan Smith said it had risen by 50% over the last five years, fuelled by changes made under Labour.
Responding to the criticisms, he said since November 2008, private rents had fallen by 5% - but the Local Housing Allowance (LHA) - used to calculate housing benefit - had risen by 3%.
Mr Duncan Smith said: "LHA has now run its unaffordable course and we must turn it around. It fuelled a landlords' charter to raise rents and has made housing more expensive for the whole population. In parts of London frankly, you can only live in parts of central London if you are on housing benefit or you are very wealthy - they have actually, you could argue, socially cleared parts of London of working people who are trying to earn a living."
The aim of capping housing benefit was to get rents down, he said: "My department is 40% of all rented housing in Britain. What we do has a massive effect on the marketplace."
Mr Duncan Smith dismissed claims the policy could end up costing £120m in temporary housing costs as "ludicrous" and accused Labour MPs of trying to "frighten" the public.
He dismissed suggestions the poor would be pushed out of London - pointing out that those in the 790,000 council and social homes in London would not be affected: "London has social housing embedded in its heart and that's not going to change."
He also said that he believed most people would not be affected by the proposed 10% cut in housing benefit for those on jobseekers' allowance for more than a year - because 90% of unemployed people found work within 12 months and because of plans for a "workfare" programme for the unemployed.
The housing benefit bill has doubled to £20bn in the past 10 years and the government, which estimates its proposals will save £2bn a year, says it is not right for claimants to be able to live in properties that people on average incomes could not afford.