A quarter of disabled people surveyed by a leading charity fear they will lose their job as a result of the coronavirus pandemic.
A poll by Scope found many disabled people fear their employer cannot make their workplace safe for them to return.
The disability equality campaign wants the government to prioritise disabled people in its recovery plans.
The government says it is committed to helping disabled people return to work.
Around 30,000 people have signed a letter to the prime minister.
Scope says disabled people have been "hardest hit" by the pandemic.
It asked 874 adults with a health problem or disability who are in employment about their views on returning to work.
It found 41% feel anxious about going back to work, while almost half are worried about using public transport to get to their workplace.
87% fear others will not respect social distancing rules, putting them at risk.
Figures released by the ONS in July showed that two-thirds of those who had died from coronavirus were disabled.
The open letter, addressed to Prime Minister Boris Johnson, points to "a looming recession and disabled people at the sharp end of poverty," and asks the government to "take urgent action."
With its survey showing 59% of disabled people felt forgotten by the government during the pandemic, the charity says the effect "is taking its toll."
It says disabled people and their families have "disproportionately faced loneliness and worsening mental health."
"Shielding may be pausing, but for millions of disabled people there is no pause button on their anxieties," the letter says.
Difficulties social distancing
Steven Morris, who is blind and has a hearing impairment, is concerned about how he will return to work as a charity policy and campaigns officer.
He needs "a lot of support when outside my home" so is particularly worried commuting.
He explains: "I need guiding to be able to get on and off trains.
"While I know that staff are supposed to assist with this, with appropriate precautions in place, I have heard a number of accounts of staff not being aware of this and refusing to assist due to the requirements of social distancing," he says.
He is worried about other passengers too, pointing to "attitudes I may encounter from other members of the public if I were to inadvertently enter their space."
Because he cannot see, he relies on other people to keep their distance, and worries that if this does not happen he will risk his health as well as his colleagues'.
He says he is unlikely to go back to the office until social distancing rules are relaxed.
"I don't feel that the government is taking my concerns as a disabled person into account at all," Steven says.
"I'm concerned that the push to get people back into their workplace could have a negative impact on disabled people."
National Disability Strategy
The letter calls for disabled people to be included in "every plan and change in regulations around the easing of lockdown from the start," and for the government to "prioritise the vital National Disability Strategy," ensuring "it provides a clear plan to mitigate existing inequalities the pandemic has further magnified."
In the last Queen's Speech, the government committed to publishing the strategy - which aims to improve disabled people's access to opportunities.
Scope's letter highlight's the prime minister commitment at the beginning of 2020 for government departments to act on the inequalities faced by disabled people.
"It is vital now that government delivers on this ambition," it says.
"Economic recovery is key, and disabled people must be included in this," it says, including measures to help them back into work.
"The commitment to tackle the disability employment gap set out in the Conservative manifesto must be upheld" it says.
It stresses that not everyone who was previously employed will be able to work in the circumstances of the pandemic and urges the government to "guarantee a safety net which supports rather than punishes" disabled people who are unemployed.
"Before coronavirus, life for disabled people and their families was tough. The government's recovery plan has forgotten about disabled people, and risks making their lives even tougher.
"An equal society shouldn't treat millions of disabled people as an afterthought," the letter says.
A Work and Pensions department spokesman said the government was committed to supporting disabled people on safely returning to work and ensuring they "have a say in determining our forthcoming national strategy".
"Huge efforts have been undertaken by employers to make workplaces COVID-secure and we want to see employers and employees discussing their working arrangement to ensure individual needs are taken into account," he said.
"We are publishing advice and guidance for disabled people, making £3.7 billion available to local authorities to help address pressures on local services including adult social care, and pledging £750 million to ensure charities can continue their vital work."