David Cameron's enterprise adviser Lord Young has quit after claiming most Britons "had never had it so good".
The Tory peer said many people had gained from low interest rates during this "so-called recession", remarks Mr Cameron said were "unacceptable".
No 10 said Mr Cameron had accepted the peer's resignation from the unpaid advisory role he took on last month.
Lord Young, an ex-cabinet minister, made the move after reflecting on media coverage of his remarks, it added.
It followed criticism of his remarks about the impact of the recession and future spending cuts and calls for his departure by Labour.
Labour leader Ed Miliband said Lord Young's initial appointment reflected badly on the prime minister as the peer was "out of touch with people".
He told BBC News: "Lord Young is right to go, I think his remarks are frankly disgraceful and many of the people who are struggling up and down this country with the consequences of the recession that we had, the consequences of the spending cuts that we are seeing, will be insulted by his comments".
He added: "Part of what this government has to do is demonstrate they do understand the consequences of the decisions they are making and the way people are struggling every day in our economy.
"I'm worried that people like Lord Young indicate a government that doesn't understand that."
The 78-year old peer, who served in Margaret Thatcher's cabinet in the 1980s, was named peer of the year at the Spectator magazine's annual awards this week, before his comments had been reported, and his recent report for Mr Cameron on health and safety was highly praised by the prime minister.
At a lunch with a Daily Telegraph journalist, Lord Young said that for "the vast majority of people in the country today, they have never had it so good ever since this recession - this so-called recession - started."
He pointed to the savings that "most people" had made on their mortgages as a result of interest rates remaining at 0.5% for the past 18 months.
He also forecast that annual job losses of 100,000 over the next four years, due to coalition spending cuts, were "within the margin of error".
Lord Young immediately apologised for the offence caused by what he said were "insensitive and inaccurate" remarks.
Explaining his decision to step down in a letter to the prime minister, Lord Young said: "In view of the reaction to the reporting of the interview I gave earlier this week, I feel that it would be right to resign forthwith from my position as your adviser."
He added: "I am a very strong supporter of you and the coalition government....You should know that you will continue to have my support in every way."
Before his resignation had been announced, David Cameron said his adviser would be doing "a bit less speaking" in the future.
"Obviously he is extremely embarrassed," Mr Cameron said of the remarks during a visit to Cornwall. "He was very quick to retract completely what he said. It was unacceptable."
Lord Young told the newspaper that the government's spending cuts, totalling more than £80bn over four years, would just take state spending levels back to what they were in 2007 - a time, he said, when people were "not short of money".
"Now, I don't remember in '07 being short of money or the government being short of money," he said.
"So, you know, I have a feeling and a hope that when this goes through, people will wonder what all the fuss was about.
"Of course, there will be people who complain, but these are people who think they have a right for the state to support them."
The Unite union said Lord Young was talking "Thatcherite claptrap" and his comments showed how divorced the government was from the problems facing workers and families across the country.
But former Conservative Cabinet minister David Mellor said Lord Young, a personal friend of his, had been badly treated.
The Tory peer had apologised after acknowledging that his remarks "were not as balanced as they ought to have been", Mr Mellor told Radio 4's PM programme, adding that he should have been allowed to stay in his job.
The former MP criticised David Cameron's handling of the matter, saying it "was like a headmaster reproving a disorderly pupil when, if anything, it might well be thought it could be the other way round".
"For the government to have lost David Young's services is a mistake," he added.
"As the political climate for the coalition darkens and as controversy gets whipped up over a whole lot of what they are trying to do the prime minister should be more careful not to cut off at the knees people who are only trying to do their best to serve him because there will be plenty of others who will blunder into these traps for the unwary.
"The prime minister needs his friends about him, not to cast them adrift the moment they say something he finds even remotely embarrassing."
Lord Young, a former successful businessman, was advising the government on how best to support small firms.