Stephen Crabb has resigned as work and pensions secretary following newspaper allegations he sent suggestive text messages to a young woman.
Mr Crabb will be replaced by Damian Green as new Prime Minister Theresa May completes her cabinet team.
The Preseli Pembrokeshire MP, promoted to the post in March, resigned "in the best interests of my family", he said.
Meanwhile Alun Cairns, MP for Vale of Glamorgan, has kept his job as Welsh secretary.
Justice Secretary Michael Gove and England's Education Secretary Nicky Morgan are leaving the cabinet.
Culture Secretary John Whittingdale is also not returning to government.
Mr Crabb, who was one of several contenders for the Tory leadership who lost out to Mrs May, said: "Over the last two years I have had the huge privilege to serve in the cabinet.
"After careful reflection I have informed the prime minister today that, in the best interests of my family, I cannot be part of her government at this time.
"I am grateful to my whole team for their hard work and encouragement. I look forward to supporting the government's One Nation vision from the backbenches."
Before taking over as work and pensions secretary, following Iain Duncan Smith's dramatic resignation over benefit cuts, Mr Crabb was Welsh secretary.
He joined the cabinet by taking up that post in July 2014, having previously been a Wales Office minister and government whip.
Tory MP Glyn Davies said Mr Crabb still has a big contribution to make.
"I'm disappointed. Stephen's an incredibly capable man," he said.
"In part because of his background but also because of his competence and his calmness, I think he made government, and he made the people, take Welsh politics seriously.
"I fully expect Mr Crabb to be back in government certainly within my period, and chances are that's only going to be four years," he said.
Mr Cairns said he was "thrilled" to be staying at the Wales Office as secretary of state.
He said: "We have a big package of measures to deliver including the Wales Bill, ensuring Welsh companies have the help they need to prosper ahead of leaving the European Union and keeping our jobs market buoyant."
Analysis by David Cornock, BBC Wales parliamentary correspondent
The political problem for Stephen Crabb was that he launched his leadership bid only two weeks ago based on values.
He stressed his family roots, he is a church-goer and there were allegations of hypocrisy.
It was hard to square the values he put forward at that campaign launch with what he was accused of doing in the Times and other newspapers.
He has now decided to take a break from government.