Women should not be sent to prison and should instead serve community sentences, according to a new report by the Women's Justice Taskforce.
The focus should be on health, housing and treatment for drug addiction to reduce reoffending, its report said.
It called for a director of women's justice to be appointed to provide "clear leadership and accountability".
The Ministry of Justice (MoJ) welcomed the report and said it was carefully considering the recommendations.
The taskforce, formed of senior police officers, magistrates, economists and penal reformers, was set up last year by the Prison Reform Trust.
Its report called for the closure of women's prisons to be accelerated, a cross-government strategy to divert women from crime and greater focus on schemes to tackle mental health needs.
Investing in community sentences instead of prison places could reduce offending and would be more cost-effective in the long-term, the report said.
Last month, Justice Secretary Kenneth Clarke's released plans to keep prison for the most serious offenders and cut the number of jail terms of under 12 months.
He has said he wanted to see a "rehabilitation revolution" which would involve "extremely serious changes" to sentencing and result in thousands of offenders avoiding jail.
In a report released on Monday, the taskforce said it was "a timely opportunity to look again at how women's justice is delivered".
More than 4,100 women were in prisons in England and Wales last week, up from 1,800 15 years ago.
The Chief Constable of Devon and Cornwall Police Stephen Otter, a member of the Prison Reform Trust which led the taskforce, said the case for keeping women out of prison was "very compelling".
"There's very clear evidence that this works for women but it also works for their family," he told BBC Radio 4's Today Programme.
Two-thirds of women sentenced to less than 12 months in jail are reconvicted within a year of their release, he said.
He said the cost of a prison place was £56,000 a year whereas the cost of a place on an intensive community order was between £10,000 and £15,000.
The government needed to take a "bold step" which would "keep the public safer", he said.
The report said: "It is true that many women end up in prison for low-level crime as a result of repeat offending and a failure to respond to non-custodial alternatives.
"Better national provision of women-focused community disposals could result in more successful completion of community orders, lower rates of reoffending and fewer women being sent to prison."
Fiona Cannon, chairwoman of the taskforce, said: "Instead of a punishment of last resort, women's prisons are now seen as stop-gap providers of drug detox, social care, mental health assessment and treatment and temporary housing - a refuge for those who have slipped through the net of local services.
"It should be possible to reform women's justice so that, with improved national and local co-ordination, better accountability and reinvestment of resources, the Government can achieve a measurable reduction both in offending by women and the women's prison population."
Roma Hooper, director of Make Justice Work, which campaigns to reform short-term prison sentences, said: "The increasing incarceration of women is a disgraceful situation which must be challenged."
"Few sensible politicians would support locking up the thousands of women - who receive the shortest sentences, at the highest cost to the state and to their families, and make up a tiny proportion of overall crime in this country."
He said community alternatives to a "short, sharp, shock" for women and their children had better success rates.
A spokesman for MoJ said: "We are continuing to develop policies that address the needs of women offenders.
"At the forefront of the justice system is the development of community based women's services, which aim to address the underlying reasons for many women's offending, such as drug and alcohol addiction, mental health and their often long histories of domestic violence and abuse.
"This work will continue."