Stephen Crabb replaces Iain Duncan Smith
Stephen Crabb has been appointed as the new work and pensions secretary, after Iain Duncan Smith resigned on Friday.
Mr Duncan Smith had said the latest planned cuts to disability benefits were "not defensible" in a Budget that benefited higher-earning taxpayers.
David Cameron said he was "puzzled and disappointed" that Mr Duncan Smith had decided to go when they had agreed to have a rethink about the policies.
Mr Crabb said there were some "very challenging decisions" to focus on.
The Preseli Pembrokeshire MP, who is being replaced by Alun Cairns as secretary of state for Wales, told the BBC: "I'm obviously really pleased to take on the role.
"It's a huge privilege to be working on issues that are very close to my heart."
Mr Crabb, 43, said it was "very sad" to see Mr Duncan Smith resign "in the circumstances in which he did".
But, he added, "there's real strong unity at the heart of this government in terms of continuing our important mission with regards to welfare reform".
An ally of Mr Duncan Smith suggested he had been driven to resign by a "high-handed and short-termist political approach" to cuts by Chancellor George Osborne.
Labour leader Jeremy Corbyn said Mr Duncan Smith had "done the right thing", having "presided over some fairly appalling policies".
He said the government's latest moves to reduce disability payments were "simply not fair, not right" and it was wrong to finance tax cuts for the better off by "taking money away" from those with disabilities.
- IDS quits: Latest updates
- Who is Stephen Crabb?
- Tory tensions over benefits
- Profile: Iain Duncan Smith
- Laura Kuenssberg: Tories dangerously divided
- In full: IDS resignation letter
- In full: Cameron's reply to IDS
In his Budget on Wednesday, Chancellor George Osborne said the government would be spending an extra £1bn on the disability budget but changes to benefits announced a few days earlier had suggested the government would save £4.4bn on this by 2020-21.
They included changes to Personal Independence Payments (PIP) that sparked an outcry from opposition parties and some Tory MPs.
But on Friday, before Mr Duncan Smith resigned, a government source had said the planned changes would be "kicked into the long grass".
BBC political correspondent Vicki Young said many ministers think Mr Duncan Smith's decision was all about the European Union referendum and he was simply looking for an excuse to leave.
He is a leading campaigner for Britain to leave the EU and on a different side in the debate to the prime minister. Sources close to Mr Duncan Smith, however, say his resignation was not about Europe.
By Chris Mason, BBC political correspondent
Before he was ousted as Conservative leader in 2003, Iain Duncan Smith was dubbed "the quiet man". But there was nothing subdued or understated about the manner of his departure last night.
Instead, a zinger of a resignation letter, designed to inflict maximum damage on Chancellor George Osborne; a relationship that had long been testy, tested to destruction by this week's Budget.
By questioning, as the Conservatives' critics have long done, the Tory slogan to justify cuts - "We are all in this together" - Mr Duncan Smith knowingly provided the government's opponents with ammunition.
For a man with ambitions to lead his party, these have been difficult days for George Osborne.
This morning David Cameron woke to a vacancy in his cabinet and a wound inflicted on his government.
'Step too far'
Baroness Stroud, who worked with Mr Duncan Smith at the Department for Work and Pensions (DWP) and now heads the Centre for Social Justice think tank, which he founded, told the BBC: "He always used to say to me, 'I'm here in order to deliver reform and to protect the poorest'.
"Yesterday he felt that he could no longer protect the poorest... This is a step too far."
Conservative MP Bernard Jenkin, an ally of Mr Duncan Smith, said: "A lot of colleagues will be extremely annoyed at this explosion but they will not be surprised.
"I think the high-handed and short-termist political approach that the chancellor has tended to take to the management of other departmental budgets has tested the patience of more than one minister."
But Conservative MP Stephen McPartland, a member of the All-Party Parliamentary Group for Disability, said he welcomed the resignation and saw it as a "real opportunity to get these welfare reforms right".
He said he was "not the chancellor's biggest supporter" but believed "aggressive reforms" and an "evangelical point of view" at the Department for Work and Pensions under Mr Duncan Smith "have consistently failed disabled people".
Commons Leader Chris Grayling told the BBC Mr Duncan Smith had done "some really good work" on welfare reform and he was "surprised" and "very sorry" to see him go.
Asked if this was really about Europe, Mr Grayling - who like Mr Duncan Smith is campaigning for the UK to quit the EU, in opposition to the prime minister - said: "I don't think this makes any difference at all to the referendum campaign."
Labour MP Frank Field, who chairs the Commons work and pensions committee, said: "Behind this is a much, much bigger drama... What he crucially cared about was the balance of resources going to families and children and older people.
"The pensioner element, the biggest part of the Budget, was safeguarded and in fact increased... all these cuts were on people of working age and it's this point that Iain thinks the social contract between generations is being broken."
In his resignation letter, Mr Duncan Smith said: "I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers."
Replying to Mr Duncan Smith, Mr Cameron said there had been collective agreement between "you, No 10 and the Treasury" that "the increased resources being spent on disabled people should be properly managed and focused on those who need it most".
"[On Friday] we agreed not to proceed with the policies in their current form and instead to work together to get these policies right over the coming months.
"In the light of this, I am puzzled and disappointed that you have chosen to resign."
Guto Bebb becomes under secretary of state at the Wales office, replacing Mr Cairns, the MP for Vale of Glamorgan.