Almost 90% of councils in England no longer offer social care to people whose needs are ranked low to moderate, the Association of Directors of Adult Social Services (Adass) has said.
The group is warning cuts are making the care system "unsustainable".
The government says councils have been given an extra £1.1bn to help protect social care this year.
But charities say hundreds of thousands of people are struggling without help.
When someone applies for social care, their needs are determined as either critical, substantial, moderate or low.
In recent years the number of councils able to help those at the lower end of the scale has gone down as they struggle to balance their budgets.
In 2010-11, Adass says 72% of councils in England only offered help with care to adults with substantial or critical needs. The association says that figure has now risen to 89%.
Left on his own
Retired miner Cyril Gillam lives in a council bungalow in Grantham, Lincolnshire. The 95-year-old widower needs help to wash, cook and clean. He eyesight and hearing are failing and he can only walk with the help of a frame.
Until a year ago he was getting home help visits twice a day. But after breaking his leg last August and spending time in hospital, his local council provided help for a number of weeks before telling the family he was no long eligible for council care.
His son-in-law Keith Harrison says: "They just dropped him like a hot potato and he was left on his own effectively. They couldn't resurrect his help."
Cyril's family challenged the decision but the council said Cyril would have to arrange and pay for his own home help visits with his £80 care allowance benefit payment.
Mr Harrison said the family contacted a number of care agencies for quotes and found the allowance would not cover the cost of the two visits a day.
Instead, a family friend calls on Mr Gillam to help him with daily tasks in return for expenses.
Lincolnshire County Council says it has changed its eligibility criteria from "moderate" to "substantial" and that on his last assessment Cyril's needs were not substantial so he did not qualify for free care.
In a statement, the council said: "Mr Gillam declined any further assessment in April and we understood the family were managing with private and family care. We've not had any contact from the family since to express any concerns and no complaint has been received."
Adass president David Pearson says: "Adult social services and local councils have done their utmost to protect services to older and vulnerable people.
"The scale of reductions in adult care spending, which amount to some £3.5 billion in the past three years, really does raise issues concerning the long-term sustainability of our services unless new money is introduced shortly."
Richard Hawkes, chairman of the Care and Support Alliance which represents 75 charities, says the care system is "in crisis".
He added: "Population changes mean more and more people need care, yet fewer and fewer people get it, as chronic underfunding has seen a year-on-year rationing of support.
"Every day, our 75 organisations hear horror stories of older and disabled people who struggle to get the support they need to simply get up, get dressed and get out of the house.
"This is also putting unbearable pressure on family carers.
"Our survey shows the public has lost confidence in the current system. It shows care, along with health, is where the public want the Government to invest more."
A Department of Health spokesperson said it had invested in social care.
"We have given an extra £1.1bn to councils to help protect social care services this year - that's on top of additional funding in recent years," he said.
"Councils are ultimately responsible for deciding how to spend their budgets but with a growing ageing population we know that we all need to work differently.
"The Care Act and our £3.8bn Better Care Fund will focus resources on helping people to live independently for as long as possible, which can save money and prevent people from needing more support."
This report will be broadcast on Sunday, 14 September 2014 on BBC Radio 5 live.
You can listen to other editions of 5 live Investigates on the BBC i-Player.