More than a million people in Great Britain now live at least a mile from a bus stop with a regular service, BBC research suggests.
The Campaign for Better Transport (CBT) said cuts to services had left some people "trapped in their homes".
Bus pass use by older and disabled people in England has fallen by almost a fifth in a decade.
The government has promised £5bn over five years for buses and cycling.
However, Department for Transport (DfT) figures show local and central government support for buses has fallen by £800m a year over 10 years.
Analysis of bus route data for the BBC's Panorama showed 550,000 properties, equivalent to about 1.3 million people, are at least 2km (1.2 miles) from a bus stop with a service calling on average four times a day.
'Your heart sinks when you need to get to the doctor's'
Marion Crawford, from Lower Denby, West Yorkshire, said her local bus stopped about seven years ago, leaving her reliant on charity-run services.
"I'm not benefitting from my bus pass at all," she said.
"I don't know what I'd do without the community bus. I'd have no way to go grocery shopping or to the doctors, I'd be reliant on friends."
The 92-year-old said: "When you go to the doctor and they say they want to see you again, your heart sinks, you think 'how am I going to get here?'"
Paul Jones, chief officer of the Denby Dale Centre, which provides a charity-funded bus service, said: "Our members tell us some bus routes do not actually reach their villages any more and they feel left alone."
The charity has received grant funding from the government and the National Lottery for its services.
Judith Lawrence, 75, said she had not left her village of Helperby, North Yorkshire, at all this year because of a lack of bus services.
No private operators run routes through the village, so the local council provides a minibus to take residents to a nearby town twice a week.
"I've got a bus pass but no bus," Ms Lawrence said.
"It's almost like a discrimination against the people in rural areas with low incomes who can't afford to take a taxi somewhere."
The lack of buses is also affecting people trying to get to work.
Construction worker Marti Blagborough said buses between his home in Farnley and Leeds city centre are often delayed or do not turn up.
He said: "If there's no buses there's no work, because we can't get there.
"You have to trust them, I guess you've got no other choice.
"If I lived locally I'd have the option of maybe bicycle to work and stuff like that but working further away it's the only option."
Bus use has declined over the past decade along with a fall of about 10% in the number of miles covered each year, but it varies across Great Britain.
According to annual DfT figures, the areas with the most steady declines in estimated miles travelled by bus include Blackburn with Darwen, which has seen a 42% reduction since 2014; Stoke on Trent (41% reduction); and North Yorkshire (41%).
Martin Kelly, Blackburn with Darwen Council's director of growth and development, said the authority had invested in public transport but "several travel companies in the area have stopped trading or have reduced the services due to falling passenger numbers".
He said other factors included austerity cuts to council funding, changing employment patterns and "lower demand for traditional shopping patterns".
"Our figures do seem to have stabilised, however, and we have recently been granted funding from the DfT which should allow us to increase certain services," he added.
Stoke-on-Trent City Council said commercial operators' fares increase when usage falls, creating "a vicious circle". The council is bidding for funding from the government to improve bus reliability and provide more services.
'Reliant on goodwill'
The government funds free bus passes for pensioners and disabled people in England but the number of journeys made with them has fallen by 18% since 2010.
The DfT said part of this decline was due to the fall in pass holders because the age at which people can receive them has risen in line with changes to the state pension age.
Darren Shirley from the CBT said: "We're seeing people who are trapped in their home, essentially."
He said many people left without a bus service would be "reliant on the goodwill of neighbours" to get around.
On average, pensioners and disabled people in England took 26 fewer bus journeys a year on their passes than they did in 2014-15, DfT data showed.
In Somerset the number of concessionary journeys taken was down by 45%.
A spokesman for Somerset County Council said it had reduced spending on concessionary fares for "a variety of reasons, including reductions in both subsidised and commercial bus services" and the collapse of a bus operator in 2016.
He added that the council continued to "meet and exceed" its statutory duties.
A spokeswoman for the DfT said: "Buses are crucial to communities, providing key links to work, school, shops and family and friends.
"We've pledged £5 billion to overhaul bus and cycling links, which is on top of the significant £220 million investment we've already made to make buses more reliable and convenient.
"We're also publishing a national bus strategy which will help transform local transport services in every region across the country."
- If you are in the UK, you can watch Panorama: Britain's bus crisis on Monday 16 March at 20:30 on BBC One or catch up afterwards on iPlayer.
Reporting team: Will Dahlgreen, Rob England, Daniel Wainwright