Jailing offenders indefinitely without providing proper access to rehabilitation courses is a breach of human rights, European judges rule.
The European Court of Human Rights blamed "lack of resources" for delays in three men doing courses before being considered for release.
They have been awarded between £12,000 and £16,000 in compensation and costs.
The government, which has scrapped the sentences, says it will appeal the court's decision.
Justice Secretary Chris Grayling told the Commons: "I am very disappointed by the decision. It is not area where I welcome the court seeking to make rulings."
There are more than 6,000 prisoners in England and Wales serving such sentences.
More than 3,500 are over their original tariff, that is the years specified by the original judge, and many of those could now be entitled to compensation.
In 2005, the three men were given indeterminate imprisonment for public protection (IPPs).
These sentences, introduced by Labour in 2003, were for offenders where it was considered they needed to be held until it was clear they were no longer a threat, even if that meant holding them beyond their tariffs. But last year the government announced they were scrapping them.
Offenders serve a minimum jail term set by a judge, after which they may apply to the Parole Board for release, which it approves only if it regards them as safe to rejoin the community.
At the European Court of Human Rights, the men argued their right to liberty has been breached because they could not get on the courses to show how their behaviour had changed. Earlier they had their cases rejected by the Law Lords.
Brett James, jailed for two years for unlawful wounding with intent, had previous convictions for battery, common assault, affray, disorderly behaviour, racially abusive behaviour and assault occasioning actual bodily harm.
Nicholas Wells, who is back in jail after being recalled in February 2010 for breaching the conditions of his release, was jailed for 12 months after being convicted of the attempted robbery of a taxi driver.
Jeffrey Lee was jailed for nine months after being convicted of drunkenly causing criminal damage to a flat in which his ex-wife and children were present. He had previous convictions for assault occasioning actual bodily harm and criminal damage.
The European judges said: "It is clear that the delays were the result of a lack of resources."
They said the inadequate resources, "appeared to be the consequence of the introduction of draconian measures for indeterminate detention without the necessary planning and without realistic consideration of the impact of the measures".
The length of the delays in the applicants' cases was "considerable," with them left in prisons without the necessary programmes for around two and a half years, the judges said.
The court also ruled the government must pay a total of almost £14,000 in damages and almost £30,000 for costs and expenses to the three men.
A Ministry of Justice spokesman said: "Public protection will not be put at risk - the judgment does not find that indeterminate sentences are unlawful, and will not mean prisoners currently serving IPP sentences will have to be released.
"The government has already announced that the complex IPP system will be replaced by a new regime of tough, determinate sentences. This will see more dangerous criminals given life sentences, and others spending longer periods in prison, with tough licence conditions on release."
Juliet Lyon, of the Prison Reform Trust, said Mr Grayling should review the 3,500 people still serving over their tariff.
"It is shaming to have so many people locked up in our prisons, not for what they have done but for what they might do in the future," she said.
"Many of these prisoners are condemned to years of uncertainty during which time they must somehow prove, from the confines of a bleak overcrowded jail, that they no longer present a risk to the public."
The ruling does not become final for three months, pending appeals.