Dragons' Den star Theo Paphitis has said Lord Young's comments that most people have "never had it so good" were "technically correct".
Lord Young resigned from his position of enterprise adviser to the government over his remarks about the recession.
Mr Paphitis also told BBC Radio 5 Live's Stephen Nolan programme he was concerned that "some half-wit" would replace Lord Young.
He was "a very experienced businessman and politician", he said.
"What has happened in this recession has never happened before," he added. "In the last recession, interest rates were up to 15% so, if you had a mortgage, all your income went on paying it.
"Now our mortgages have gone down because interest rates have gone down to practically zilch, so of course we have had more disposable income.
"So, technically, those of us who have a job and are still in work, and not worried about losing our job, have actually been better off, so he [Lord Young] was technically correct. So why did we fire him?
"My worry is they are going to put some half-wit in now that Lord Young has gone. He was a very experienced businessman and politician as well."
At a lunch with a Daily Telegraph journalist, Lord Young said that for "the vast majority of people in the country today, they have never had it so good ever since this recession - this so-called recession - started".
He quickly apologised for his comments, but resigned on Friday from the unpaid advisory role he took on last month.
Downing Street called his remarks offensive and said Mr Cameron was "unimpressed".
Speaking to Sky News on Sunday, the prime minister said Lord Young's resignation had been "the right outcome" because "in politics what you say does matter".
He said: "I'm sad about it because... he did a brilliant report on health and safety, cracking a problem that's bedevilled governments for a long time."
Labour leader Ed Miliband said Lord Young's initial appointment reflected badly on the prime minister as the peer was "out of touch with people".