The "shocking" case of a man with learning disabilities and terminal cancer has led to a call for the prime minister to appoint a commissioner for learning disabled people's rights.
Ian Shaw, 34, spent nine years in secure hospitals before being moved into community care last year.
The cancer was found and his parents say the units should have spotted it.
Sir Stephen Bubb, author of two reports on secure units, has told Theresa May Ian's case highlights ongoing failures.
Sir Stephen told the BBC that Ian's story was "all too typical".
"It has led me to believe that institutional care is at root abusive and we must close these institutions."
In his letter to Theresa May, he says that half of all deaths of people with a learning disability in 2015 were recorded as avoidable, "compared with 23% for the general population".
He urges her to look again at a recommendation he made in his second report, last year, that the government should set up an office of a Commissioner for People with Learning Disabilities.
The aim would be to uphold the rights of learning disabled people similar to the way Children's Commissioners uphold those of children.
"People with learning disabilities are among the most vulnerable in our community. They need support and they need protection. They and their families face immeasurable odds in tackling the system that often neglects them," he writes.
Sir Stephen wrote his reports after the Winterbourne View scandal, which was exposed by BBC Panorama six years ago.
Six care workers at the private secure hospital were eventually jailed for the neglect or abuse of patients and five others were given suspended sentences.
In December 2012, the government announced plans for a "dramatic reduction" in the number of people with learning disabilities kept in hospitals in England, with more appropriate care provided closer to people's families.
But five years on, about 2,500 people are still in secure assessment and treatment units.
Ian, who also has autism and epilepsy, spent nine years in secure hospitals before being moved into community care last year.
He is now living at the family home in Essex and he draws comfort from having his favourite song on repeat.
It is the 1980s hit Mad World and his parents, who look after him, are finding the words particularly hard to bear: "The dreams in which I'm dying are the best I've ever had."
Ian's disability means he is unable to speak for himself.
His mother, Jan, says the family has been told there is no treatment as his testicular cancer is too advanced.
"Because it had been there a long time and they couldn't treat it because it would be too much, the treatment just wouldn't work. the disease had gone too far," she explains.
Ian was sent to the first of three secure units in 2007 when his behaviour became increasingly challenging.
The family say, pain from his conditions caused him to throw things or bang his head against the wall, scarring himself.
His mother believes "the secure units were the wrong places" but it proved very difficult to get him out.
Jan raised numerous concerns about levels of medication which she believed were too high.
The family kept records of the times Ian was restrained and fought to have him transferred to a supported home in the community.
His cancer was found soon after he was finally moved and the family believes the secure unit failed to spot and investigate early signs of the disease.
Bernadette Adams, of the charity Autism Inclusion, who supported the family in their battles, believes Ian has been failed.
"Jan has been saying for many, many months that Ian was in pain or Ian had infections and she was on many occasions, if not all occasions, just ignored."
The call for a Learning Disability Commissioner has been welcomed by charities such as the Challenging Behaviour Foundation, which works to support families of people like Ian.
Chief executive Vivien Cooper called the suggestion helpful but added that it needed to be part of a "clear strategy, resources and implementation framework... so that children and adults with a learning disability can exercise their rights and enjoy the same life opportunities as anybody else".
The Department of Health said ministers had no plans for a learning disability commissioner, saying that the government had ensured considerable independent scrutiny of learning disabilities in recent years, including by Sir Stephen.
A spokeswoman said: "For too long, people with learning disabilities have not been treated equally by the health service and we're determined to change this.
"In recent years we've made significant improvements by closing inpatient facilities and moving towards personalised community-based care, supported by rigorous independent inspections to stamp out poor care and abuse, to give people the support they deserve."
NHS England said it had set out a clear programme to improve services for people with learning disabilities who display challenging behaviour.
Key aims include reducing unnecessary hospital admissions and lengthy hospital stays as well as giving patients and their families more choice in their care, alongside more care in the community and personalised support.