Prisoners held indefinitely after serving their minimum term or tariff should not have to prove it is "safe" to release them, new Parole Board chairman Nick Hardwick has said.
Various factors make it "incredibly difficult" for some inmates on Imprisonment for Public Protection sentences to find such proof, he said.
He wants new criteria for freeing IPP prisoners in England and Wales.
The Ministry of Justice said the suggestion had been "taken on board".
IPP sentences were introduced by Labour in 2005 as a way of stopping the release of dangerous prisoners.
But courts were banned from imposing any more IPP sentences in 2012 amid concerns they were being used to hold people for periods which their original offence did not warrant.
In March, 4,133 IPP prisoners continued to be detained, the majority of whom had been convicted of "violence against the person", sexual offences or robbery.
'Festering in prison'
The Parole Board can approve a prisoner's release after the minimum term - the "punishment" part of their sentence - but only if it is satisfied it is not necessary to hold the inmate in the interests of public protection.
It means the prisoner has to prove they do not present a risk and can be safely managed in the community.
In March, about 80% of IPP prisoners - 3,347 - had already served their minimum term but were still locked up.
In his first interview since taking up his post in March, Prof Hardwick told the BBC that procedural delays, problems accessing offending behaviour courses and finding suitable accommodation made it "incredibly difficult" for some IPP prisoners to prove that it was safe for them to be let out.
"Some of them are stuck, festering, in prison long after the punishment part of the sentence," he said.
Ministry of Justice figures show more than 500 IPP prisoners given tariffs of less than two years were still in prison five or more years later.
"Once it gets to that point, they stop making progress and they start going backwards," said Prof Hardwick.
"So this is, I think, a blot on the justice system and I'm very keen we can do something about it."
He said Liz Truss, the new justice secretary, should consider activating Section 128 of the Legal Aid Sentencing and Punishment of Offenders Act 2012.
The clause allows the justice secretary to alter the test which the Parole Board has to apply when releasing prisoners.
Both houses of Parliament would have to agree to the change, but fresh legislation would not be required.
"There are legislative options that will enable us to change the risk test so it's more about 'is there proof that they're dangerous rather than proof that they're safe?' and there are some other measures that can be taken... to try to cut into that group," Prof Hardwick said.
The former Chief Inspector of Prisons said there were three categories of IPP inmate who would benefit most: Those on very short tariffs but still in custody; prisoners held beyond the maximum sentence for the offence they had committed; and offenders who were too frail or elderly to pose a danger.
The Parole Board is also trying to cut the backlog of prisoners awaiting decisions on their release, by hiring more parole panel members and dealing with cases more efficiently.
Prof Hardwick said it was "crazy" to be paying out compensation to inmates held in custody because their cases were delayed due to a lack of resources.
In 2015-16, there were 463 damages claims lodged, five times the number the previous year, with £554,000 paid out in compensation, compared to £144,000 the year before.
"It's not a good use of taxpayers' money," Prof Hardwick said.
"It would be much better to put the money into ensuring that the system is working efficiently so that people get dealt with fairly and get out when they're supposed to and when the courts intended."
The Ministry of Justice said: "The chair of the Parole Board has made a number of recommendations to improve the parole system and reduce the backlog of IPP prisoners.
"Work is ongoing within the department to address these issues and his recommendations have been taken on board".