The High Court has rejected a challenge by two charities against cutting legal aid for prisoners.
The Howard League for Penal Reform and the Prisoners' Advice Service (PAS) had sought a full judicial review, arguing vulnerable inmates would suffer.
They say they will now appeal.
Ministers said the changes, introduced in 2013, would stop taxpayers' money being used for "unnecessary legal cases" that could be handled by the prison service.
Justice Secretary Chris Grayling, who is also the Lord Chancellor, said it would mean 11,000 fewer cases - and savings of £4m - a year.
Mr Justice Cranston and Lady Justice Rafferty, sitting in London, said they could "well understand the concerns" raised by the new rules.
"But we simply cannot see, at least at this point in time, how these concerns can arguably constitute unlawful action by the Lord Chancellor," they added.
"For the time being, the forum for advancing these concerns remains the political."
The charities say Mr Grayling has imposed the cuts in England and Wales on inmates who will be "too vulnerable to complain" for "ideological reasons entirely contrary to the rule of law".
At a hearing in London earlier this month their lawyers said the changes were "unfair, irrational and inflexible".
They said prisoners' rights - and their chances of rehabilitation - had been undermined, adding that the changes would cost the taxpayer millions of pounds more in "hidden costs".
Phillippa Kaufmann QC, appearing for the charities, said the cuts stopped legal assistance for women prisoners facing reviews over eligibility for mother-and-baby units.
They also affected inmates facing segregation and placement in close supervision units, she added.
But James Eadie QC, for Mr Grayling, said arguments made in court about the "victimising and targeting of prisoners" had been addressed by Parliament.
New regulations were approved with a backdrop of a need for "financial stringency in the legal aid system because of scarce resources", he said.
And the charities could not point to any lead cases of unfairness having actually arisen, he added.
Reacting to Monday's ruling, the Prisoners' Advice Service said it was "deeply disappointed with this judgment, which fails to respond to the increased unfairness prisoners now face as a result of the latest round of legal aid cuts".
"The Court is right to say that this is a political issue," joint managing solicitor Deborah Russo said.
"However, that does not mean that it is one in which the law cannot intervene if prisoners' fundamental rights of access to legal remedies are being breached."
Howard League for Penal Reform chief executive Frances Crook said the cuts would not save the taxpayer money.
"On the contrary, they will result in increased costs as children remain in prison for longer than is necessary for want of a safe home to go to," she added.
She said the charities would "take this to the Court of Appeal as the High Court made fundamental errors in its understanding of some of the key points".
"It did not properly deal with the concerns of the Joint Committee on Human Rights that the complaints system cannot be effective in certain cases," she added.
"The court completely failed to address how unfairness would not arise in particular situations where prisoners are unrepresented."
She said these included parole board hearings where secret evidence was used against prisoners or "other cases which turn on expert evidence that cannot be commissioned without legal representation and funding".
Announcing the plans last year, Mr Grayling said: "I have been appalled that taxpayers pay millions of pounds every year supplying lawyers for prisoners to bring unnecessary legal cases.
"The vast majority of these types of complaint can and should be dealt with by the prison service's complaints system.
"After years spiralling out of control, the amount spent on legal aid for prisoners is being tackled."