David Cameron says he is "puzzled and disappointed" after Iain Duncan Smith quit as work and pensions secretary.
Mr Duncan Smith said the latest planned cuts to disability benefits were "not defensible" in a Budget that benefited higher-earning taxpayers.
In response, Mr Cameron said they had "collectively agreed" on the proposals, before deciding on Friday to have a rethink to "get these policies right".
Stephen Crabb has been named as Mr Duncan Smith's replacement.
By Chris Mason, BBC political correspondent
Before he was ousted as Conservative leader in 2003, Iain Duncan Smith was dubbed "the quiet man". But there was nothing subdued or understated about the manner of his departure last night.
Instead, a zinger of a resignation letter, designed to inflict maximum damage on Chancellor George Osborne; a relationship that had long been testy, tested to destruction by this week's Budget.
By questioning, as the Conservatives' critics have long done, the Tory slogan to justify cuts - "We are all in this together" - Mr Duncan Smith knowingly provided the government's opponents with ammunition.
For a man with ambitions to lead his party, these have been difficult days for George Osborne.
This morning David Cameron woke to a vacancy in his cabinet and a wound inflicted on his government.
In his Budget on Wednesday, Chancellor George Osborne had said the government would be spending an extra £1bn on disability but changes to disability benefits announced a few days earlier had suggested the government would save £4.4bn by 2020-21.
They included changes to Personal Independence Payments (PIP), which will replace Disability Living Allowance (DLA) in January 2017, that were expected to save £1.3bn a year and sparked outcry from opposition parties and some Tory MPs.
On Friday, prior to Mr Duncan Smith's resignation, a government source said the planned changes would be "kicked into the long grass".
Baroness Stroud, who worked with Mr Duncan Smith in government and now heads the Centre for Social Justice think tank, which he founded, told BBC Radio 4's Today programme: "Iain came into government... in order to deliver a social justice agenda, a reform agenda of the welfare state.
"He always used to say to me, 'I'm here in order to deliver reform and to protect the poorest'.
"Yesterday he felt that he could no longer protect the poorest... This is a step too far."
Defence Secretary Michael Fallon told Today he was also "disappointed" by Mr Duncan Smith's resignation but the welfare reform programme would "go on".
Frank Field, who chairs the Commons work and pensions committee, said: "Behind this is a much, much bigger drama of which Iain was the driver... What he crucially cared about was the balance of resources going to families and children and older people.
"The pensioner element, the biggest part of the Budget, was safeguarded and in fact increased.
"And therefore all these cuts were on people of working age and it's this point that Iain thinks the social contract between generations is being broken."
In his reply to Mr Duncan Smith's resignation letter, Mr Cameron said there had been collective agreement that "the increased resources being spent on disabled people should be properly managed and focused on those who need it most".
"That is why we collectively agreed - you, No 10 and the Treasury - proposals which you and your department then announced a week ago," he said.
"Today we agreed not to proceed with the policies in their current form and instead to work together to get these policies right over the coming months.
"In the light of this, I am puzzled and disappointed that you have chosen to resign."
Mr Cameron's letter also highlighted that they were "on different sides in the vital debate about the future of Britain's relations with Europe".
Sources close to Mr Duncan Smith, who wants the UK to leave the EU, say his resignation was not about Europe.
'A compromise too far'
In his resignation letter, Mr Duncan Smith said that "because of the perilous public finances we inherited from the last Labour administration, difficult cuts have been necessary".
"I have found some of these cuts easier to justify than others but aware of the economic situation and determined to be a team player I have accepted their necessity.
"You are aware that I believe the cuts would have been even fairer to younger families and people of working age if we had been willing to reduce some of the benefits given to better-off pensioners but I have attempted to work within the constraints that you and the chancellor set."
But he went on: "I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers."
He added: "I am unable to watch passively whilst certain policies are enacted in order to meet the fiscal self imposed restraints that I believe are more and more perceived as distinctly political rather than in the national economic interest."
BBC political correspondent Iain Watson said: "Tensions with the chancellor had been simmering for some time, but there was little warning of Friday's explosion.
"What seemed to be at its core were briefings by government sources which placed responsibility for the controversial changes to PIP firmly at Mr Duncan Smith's door, while he had in fact felt some pressure from the Treasury to deliver them."
Mr Duncan Smith was Conservative Party leader and Leader of the Opposition from 2001 to 2003, and had been work and pensions secretary since the 2010 general election.
Labour leader Jeremy Corbyn and Liberal Democrat leader Tim Farron have both called for Mr Osborne to resign.
Mr Corbyn said: "The resignation of Iain Duncan Smith reveals a government in disarray and a chancellor who has lost the credibility to manage the economy in the interests of the majority of our people."