Ministers have failed to honour a pledge to move patients with learning difficulties out of hospitals and into community care, the National Audit Office (NAO) says.
There were still 2,600 such inpatients in mental health hospitals in England in September 2014, the NAO said.
A commitment to move patients was made in the wake of the Winterbourne View abuse scandal.
Care Minister Norman Lamb said the government was stepping up efforts.
But Margaret Hodge, chairwoman of the Commons Public Accounts Committee, said the failure was "unacceptable".
Ministers pledged in December 2012 that any inpatient with a learning disability or challenging behaviour, who would be better off cared for in the community, would be moved out of hospital by June 2014.
The promise came after BBC's Panorama programme exposed abuse of patients by staff at the Winterbourne View private hospital near Bristol.
But the NAO said ministers had underestimated the "complexity and level of challenge" involved in discharging so many patients into the community.
The government's NHS reforms meant they lacked the "traditional levers" to implement the necessary changes, its report added.
It said figures from NHS England showed that by March 2014 there were still more than 2,600 inpatients with learning difficulties.
That number has remained broadly stable, according to figures from September 2014.
'Badly let down'
While NHS England set a new "ambition" in August 2014 to transfer half of the 2,600 inpatients to more appropriate care by the end of March 2015, the NAO said that so far only around 400 had been moved.
The report also showed that from the 48 patients in Winterbourne View at the time of its closure, 10 were still in hospital, 20 were living in residential care, five were in supported housing, 12 had their own tenancies, and one had died.
Ms Hodge said: "People with learning disabilities, admitted to hospitals for assessment and treatment, have been badly let down by government.
"More than two years since its response to the abuse of patients at the Winterbourne View hospital, it is unacceptable for government to have failed in meeting its core commitment to move people out of mental health hospitals and into the community."
In a joint statement, the charities Mencap and the Challenging Behaviour Foundation, said the report showed "abject failure" to tackle the issue.
Mr Lamb admitted the government had not gone far enough.
He added: "I am looking at legislative options to give people with learning disabilities and their families a stronger voice and more rights - and I'm looking at how we can increase specialised housing options, so that more people can live independently but with the support that they need."