Foreign Office minister Mark Simmonds has resigned from the government, the second to depart in less than a week.
No 10 said Mr Simmonds made the decision "several weeks ago" and said it was unrelated to the events in Gaza which led to Baroness Warsi quitting.
He will also step down as an MP at the election next year - telling the BBC he had to "put family life first".
Downing Street has announced that James Duddridge will replace Mr Simmonds as minister for Africa.
Mr Simmonds said he had agreed his departure as a minister at the time of last month's government reshuffle, saying that the PM only wanted people who were standing at the next election in ministerial jobs.
The prime minister's spokesman said Mr Simmonds had stayed on in the job until now, to chair meetings at the UN on the situation in the Democratic Republic of Congo.
Mark Simmonds at-a-glance
Educated: Worksop College and Trent University
Family: Married with three children
MP: First elected, in Boston and Skegness, in 2001
Minister: Foreign Office minister since October 2012
Mr Simmonds told the BBC there was "nothing suspicious" about his resignation and no policy problems. He said the reason he had decided to step down as an MP was related to the expenses system.
He said: "The allowances that enable Members of Parliament to stay in London while they are away from their families - my family lives in Lincolnshire in my constituency - does not allow me to rent a flat which can accommodate my family, so I very rarely see my family and I have to put family life first."
Mr Simmonds said the idea of spending another five years rarely seeing his children and staying in a different hotel room each night "fills me with horror".
The Independent Parliamentary Standards Authority said an MP with three children could claim £27,875 a year to rent a flat in London.
According to the most recent information on the IPSA website Mr Simmonds, who was paid £89,435 as a junior minister, also employed his wife Lizbeth as his office manager in 2012/13 on a salary "between £20,000 and £24,999".
Of his ministerial role, he said he was "proud" of the work he had done and said his successor had knowledge of Africa and would be able to "hit the ground running".
But Sir Alistair Graham, the former chairman of the Committee on Standards in Public Life, told the BBC that he thought the public would be "shocked to hear somebody say they couldn't manage on £90,000-a-year".
He said: "I don't think public are going to have much sympathy with his view that he couldn't make suitable arrangements to try and spend some time with his family, and carry out his public service duties."
Asked whether it might have made more sense to stay on in the job, given the current situation in Iraq and Gaza, Mr Simmonds said his departure had been agreed weeks ago.
"The prime minister understandably wanted to have a new minister coming in to replace me once I told him I wasn't intending to stand at the next election," he said.
Mr Simmonds also told the BBC that the recent suggested electoral threat to the Conservatives in his Boston and Skegness constituency from UKIP had almost led to him deciding to stay on.
For Labour, shadow foreign minister John Spellar said the resignation was "another blow to a government already reeling" from Lady Warsi's departure.
"At a time when the FCO (Foreign Office) should be focused both on the crisis in Iraq and the situation in Gaza following the recent appalling violence there, David Cameron's government instead risks looking increasingly rudderless and characterised by confusion."
In his reply to Mr Simmonds' resignation letter, Mr Cameron said he had been a "good friend" and still had "much more to give".
Baroness Warsi resigned as a Foreign Office minister on 5 August, saying the UK's policy on the crisis in Gaza was "morally indefensible".