Ex-Justice Secretary Sir Robert Buckland is calling for changes to mental health laws to stop people with autism or learning disabilities being detained in hospitals for long periods.
Last year, the BBC found that 100 people with learning disabilities and autism had been held in hospitals for at least 20 years.
Sir Robert said the "effective caging of human beings" was "out of date".
The government said it was "determined" to reduce numbers in hospitals.
A spokesperson from the Department of Health said it was "investing in community services and supporting discharges," with £90m of additional funding this year.
And Health and Social Care Minister Gillian Keegan said the government was creating an "action plan" to be rolled out as soon as possible.
Sir Robert, who has an autistic daughter, raised the subject in Parliament on Thursday, apologising for not doing more on the issue when he was on the front bench.
The South Swindon MP told the Commons 2,085 people with autism or a learning disability were in inpatient units at the end of November, and 200 of them were under 18.
He said the average length of stay was 5.4 years, but some had been detained for 20 years or more.
In September 2021, there were 3,620 reported incidents of restrictive interventions - involving either physical or chemical restraint - with 595 of these involving children, he added.
The former minister described "the effective caging of human beings" in some cases, "often in small spaces, with food being delivered by a hatch, mattresses on floors and conditions that can only be described as inhumane".
Now he wants the government to ensure better and earlier interventions to prevent cases "spiralling into crisis in the first place", as well as better community-based alternatives to inpatient detention.
Speaking to the BBC ahead of the debate, Sir Robert said the system "has to stop discriminating against people who somehow it regards as less important and that is what has happened here".
He added: "It's not hostility, it's indifference that is causing the problem."
In 2015 - prompted by an abuse scandal at Winterbourne View specialist hospital - the government pledged to reduce the number of patients in hospital settings by 35% by March 2020 by providing more help for people in their community.
However, the target was missed with numbers falling by 300 patients - a reduction of 13%.
In 2018, Professor Sir Simon Wessely produced a report after an 18-month review, calling for new rights of mental health patients detained in hospital.
His report said it should be easier for patients to challenge their treatment, including their detention.
Theresa May - the prime minister at the time - said the review would be used to make changes to legislation, but they have yet to come to fruition.
Sir Robert said the government had agreed to reform the Mental Health Act and had the right "legislative intent", but they "just need to get on with it".
"I want to give a very firm shove to my colleagues in government to crack on with the reforms that Simon Wessely and others have recommended," he added.
The minister, Ms Keegan, promised in the Commons that it was a priority for the government that all autistic people and those with learning disabilities "must receive high quality safe care", adding they would "not tolerate abuse or poor quality care".
A health department spokesman said the government's NHS Long Term Plan now committed them to reducing inpatient numbers of adults and children with a learning disability and autistic people by 50% by 2024 through investment in community services.
They added: "We are determined to continue to reduce the number of autistic people and people with a learning disability in mental health hospitals and reduce the reliance on inpatient care.
"That's why we are investing in community services and supporting discharges with £90m of additional funding this financial year."