There is no link between rising levels of imprisonment and falling crime, Justice Secretary Ken Clarke has said.
With crime having fallen in most of the Western world in the 1990s, he said the decline may have been due to economic growth and high employment levels.
Meanwhile, the Chief Inspector of Prisons for England and Wales, Dame Anne Owers, warned that "overpopulated" prisons are "increasingly brittle".
She said the government should invest in alternatives to locking people up.
Mr Clarke told judges at their annual Mansion House Dinner in London that "no-one can prove cause and effect" for why crime fell in the 1990s.
His comments come after former Tory home secretary Michael Howard recently criticised him for attacking high imprisonment rates.
BBC home affairs correspondent Danny Shaw said the justice secretary's comments appeared to be a swipe at Mr Howard, who coined the phrase "prison works" in 1993.
In his speech on Tuesday night, Mr Clarke said: "There is and never has been, in my opinion, any direct correlation between spiralling growth in the prison population and a fall in crime.
"Crime has fallen in Britain throughout a period of both rising prison populations, and throughout the same period of economic growth, with strong employment levels and rising living standards.
"No-one can prove cause and effect. The crime rate fell but was this the consequence of the policies of my successors as home secretary or, dare I gently hint, mine as chancellor of the exchequer at the beginning of a period of growth and strong employment? We will never know."
The justice secretary, who favours rehabilitation and community sentences, said crime had fallen in Canada in the 1990s after the prison population was cut by 11% and that crime did not rise significantly in Finland when similar measures had been taken there.
Dame Anne Owers' warning that prisons are now increasing brittle came in her valedictory lecture to the Prison Reform Trust on Tuesday night.
She said prisons "had become better places" but progress in rehabilitating offenders was slow because of the growing prison population.
She called upon ministers to "do things differently" in the "age of austerity" and invest in alternatives to prisons.
"We now have an inflated prison system in a shrinking state. It [is] crucial to invest in 'not prison' - both instead of and after prison," she said.
She also issued a warning over the number of inmates - one in six of the prison population - serving indeterminate sentences, saying they were a source of "increasing volatility and potential disruption".
"This is and will continue to be a significant upward driver of the prison population," she said.
A Ministry of Justice spokesman said as part of the "rehabilitation revolution", the government was working towards providing a "fit for purpose prison estate" by building new prisons and closing "inefficient and worn out places".
"The government has announced its intention to conduct a full assessment of sentencing policy to ensure that it is effective in deterring crime, protecting the public, punishing offenders and cutting re-offending," he added.