Families of people with learning disabilities and autism say they are planning to take legal action against local authorities and NHS providers over lack of provision in the community.
Debbie Evans is a mother who feels using the law is her last resort.
Her 24-year-old son Eden has been in institutions for seven years.
Over a period of five years his weight increased by 16 stone (101kg).
"He would say to me, 'You've got to get me out Mummy, you've got to get me out,'" she said.
Mrs Evans found it difficult to support her son, who has autism and a learning disability, when he lived at home in west London.
She said he had no formal education from the age of eight because there was no proper provision for him.
"It got really hard with him when he was 14; he hadn't had any education and basically he lashed out at me," she said.
"It got to the point that I couldn't breathe - it was 24 hours a day seven days a week."
She sought help and he ended up in an assessment and treatment unit.
These are meant to be short stay hospitals where patients are given a plan of care to support them back into the community, but this did not happen in Eden's case.
"He's been trapped for seven long years," Debbie said.
Human rights concerns
Emma Jones is a human rights lawyer from Leigh Day solicitors representing about five families living in England.
She believes they have a case to be answered and is exploring avenues for legal action to be taken under the Care Act, the Children's Act and the Human Rights Act.
She said: "In a nutshell, the position we are facing in this country is that there aren't enough community provisions being provided which means that people who no longer meet the criteria to be detained remain locked up because [there is] nowhere else for them to go.
"If they're locked up when they shouldn't be locked up they are detained unlawfully and that's a breach of their human rights."
Since the Winterbourne View scandal in 2011, when an undercover Panorama investigation revealed abuse of people with learning disabilities, there has been debate as to the best way to support some of the most vulnerable people in society.
Numerous reports have been published and NHS England has responded with what is described as a "far reaching" plan.
The aim is to reduce the number of people in institutional care by up to 50% over the next three years by building up the level of support in the community.
There are more than 2,500 people with learning disabilities and autism kept in institutions across England.
According to the latest Learning Disability Census, nearly a third have care plans that clearly state there is no reason for them to receive inpatient care.
A spokesperson for NHS England said: "Every case is different and patients' needs are often extremely complex, but we have been clear that hospitals should not be seen as homes.
"Where admission is deemed by clinicians to be in their best interest, patients should stay no longer than they need to.
"From this month, local areas will begin implementing a plan to ensure that the housing, care and advocacy services become available in each community to provide the high-quality alternatives to hospitals."
The families of those still in the system say that every minute, every hour is valuable and things have so far, moved far too slowly.
A social media campaign is due to launch on Monday called "Seven Days of Action".
Each day will see the release of a new story of someone's son, brother, sister or daughter far away from their families.
They are doing it, they say, to put a face to the numbers.