The Chancellor has been criticised by a disabled charity for saying high levels of disabled people in the workforce may have had an impact on productivity.
Speaking to the Treasury Committee, Philip Hammond said the UK should be "extremely proud of high levels of participation by disabled people".
But he said that may have had an impact on the UK's overall productivity.
Disabled charity Scope called for an apology, saying the comments were "unacceptable and derogatory".
Last month the government announced plans to get a million more disabled people into work within a decade.
Anna Bird, director of policy and research at Scope, said Mr Hammond's comments "fundamentally undermine the government's policy and the ambition set out by the prime minister just a week ago".
She called on him urgently to withdraw the remarks.
Office for National Statistics figures show that disabled people remain twice as likely to be unemployed as their able-bodied peers.
Labour MP Marsha de Cordova, shadow minister for disabled people, tweeted: "As a disabled person I am shocked and appalled that Philip Hammond is trying to blame me and other disabled people for the Tories' economic failure.
"He should apologise immediately for this disgraceful comment."
She said the disability productivity gap had been "stuck at 30 odd percent" since 2010 and she called for a massive investment strategy and education programmes.
Sophie Morgan, a disability rights campaigner and presenter, expressed her anger on Twitter.
This lie is a huge set back. There is NO evidence to back up his claim whatsoever. Our struggle to work - and be VALUED in society - is real and this will only make it harder. https://t.co/H1Co1bUAxt— sophie morgan (@sophmorg) December 7, 2017
She was joined by inclusion expert Mik Scarlet, who responded to Mr Hammond by documenting his journey to work.
Urh. Feeling all unproductive as I wake up @ 5.30am to leave for worl this morning. Traveling up to Birmingham to help an industry be more productive & attract more business. But as a pesky Disabled employee I must be dragging our country down. #hammondmustgo— Mik Scarlet (@MikScarlet) December 7, 2017