Stores refusing to accept cash have left consumers unable to buy basics such as groceries and medicine, according to Which?
The consumer group warned the cash system is being threatened as shops have declined payments using banknotes and coins during the coronavirus crisis.
Thousands of people have been prevented from paying with cash in recent months.
This risks excluding vulnerable people, the campaign group said.
Some shops have refused payments with banknotes and coins during the coronavirus crisis due to social distancing concerns, but this has threatened the viability of the cash system, it warned.
Thomas Scobie of Stirling, who lives on universal credit, said: "When shops started to accept only card payment it meant I couldn't buy the essentials I needed to feed myself."
He has a chronic health condition and a mental health disorder, so found the process of finding places to shop that would accept his cash "a real struggle and depressing".
"The reason I don't use a card is because I worry about the people that are able to clone cards and scam people and being on a fixed income, I simply couldn't survive if I lost any of that money," he said.
Which? is asking businesses to show greater understanding and flexibility to customers who may only be able to pay in cash.
"The rapid move towards a cashless society risks excluding the most vulnerable from being able to pay for vital products and services," said Richard Piggin, head of external affairs and campaigns at Which?.
"We're alarmed at the reports of people leaving food and medicine behind because they can't pay with cash and it underlines how important it is to have a co-ordinated approach to protecting the fragile cash system."
Natalie Ceeney, chairwoman of the Access to Cash Review, said: "Businesses who don't accept cash are saying to the most vulnerable in society: 'You're not welcome here'.
"Right now, as so many people are going through hardship and isolation, it's critical that no one gets excluded."
The government has plans to make rules to protect cash and, as part of this, the Financial Conduct Authority (FCA) could oversee that.
Which? said it supports the proposal regarding the FCA and wants to see the regulator track levels of cash acceptance.
"The government has already proposed giving the FCA responsibility for cash," said My Piggin.
"It's vital that acceptance is also treated as a priority as part of this, as commitments to safeguarding cash access will be severely undermined if people are left with nowhere to spend it."
The Access to Cash Review revealed eight million people were at risk in the UK from the demise of cash.
Nearly 2,500 people responded to Which?'s call for people to report their payment problems last month.
Two fifths of those who reported being unable to pay with cash said that they did not have access to another payment method.
Some people were able to go to another shop to buy what they wanted but almost a third were unable to buy items or services at all.
Which? said two out of five people were left empty handed when they had problems paying for groceries, while nearly a fifth had problems trying to buy medicine.
Andy Fisher in Beverley said: "I was told by one shop assistant that cash was 'a thing of the past' when I tried to buy some stationery, which made me feel uncomfortable and patronised.
"I feel that coronavirus is being used as an excuse to get rid of cash but lots of people locally still need it, such as the elderly or vulnerable."
John Howells, chief executive of cash machine network Link, said: "We can't afford to sleepwalk into a cashless society where people are left behind."
Caroline Abrahams, charity director at Age UK, said: "Many older people rely on cash and it's really disappointing that even after venturing out to do their shopping, which for some feels like a significant risk at the moment, they may then be unable to buy their essential items."