President Barack Obama has called for sweeping reforms to the US criminal justice system including curbing the use of solitary confinement and voting rights for felons.
He said lengthy mandatory minimum sentences should be reduced - or thrown out entirely.
"Mass incarceration makes our entire country worse off, and we need to do something about it," he said.
Mr Obama urged Congress to pass a sentencing reform bill by year's end.
On Thursday, Mr Obama will be the first sitting president to visit a federal prison - part of week long focus by the White House on the criminal justice system.
Speaking to a gathering of the National Association for the Advancement of Colored People (NAACP) in Philadelphia, Mr Obama discussed investments in education, alternatives to trials and prison job training programs.
US Attorney General Loretta Lynch has been tasked with reviewing the overuse of solitary confinement, Mr Obama said.
"Do we think it makes sense to lock people up in tiny cells for 23 hours a day? It won't make us safer and stronger."
The country should not be tolerating overcrowding in prisons, gang activity or rape, which Mr Obama called "unacceptable".
Criminal justice reforms have been a subject of rare agreement between Republicans and Democrats in Congress.
He noted that African Americans and Latinos disproportionately make up most of the prison population.
On Monday Mr Obama commuted the sentences of 46 prisoners, many of whom were serving time for non-violent drug offences.
"If you're a low-level drug dealer, or you violate your parole, you owe some debt to society … but you don't owe 20 years," Mr Obama said.
Mr Obama said for what the US spends on keeping people in prison per year, $80 billion, there could be universal pre-school, doubled salaries for high school teachers or free tuition at US public colleges or universities.
This week's focus on criminal justice signals a renewed bid by Mr Obama's administration to tackle what he sees as a lack of fairness in the system.
"Communities that give our young people every shot at success, tough but fair courts and prisons that seek to prepare returning citizens to get that second chance...That's what we're here to build," he said.
The last significant changes to the criminal justice system in the US came in 2013 when US Attorney General Eric Holder dropped mandatory minimum sentences for non-violent drug offenders.