Lord Coe has indicated that he has withdrawn from the race to be the next chairman of the BBC Trust.
The Tory peer, currently chairman of the British Olympic Association, was one of the favourites for the role.
But he told the Daily Mail he wanted to stand instead for the presidency of the International Association of Athletics Federations.
He said that "on reflection" he did not have enough time to do the BBC job and athletics was "in his DNA".
Lord Coe said: "I did allow my name to go forward to give myself time to properly analyse whether I had enough time to do the job to the best of my abilities.
"'On reflection, I haven't the capacity and I now want to concentrate on my current commitments and the IAAF election. As everyone knows, athletics is in my DNA."
Lord Coe, 57 - who won Olympic gold medals in the 1500 metres in 1980 and 1984 - will concentrate on his aim of succeeding Lamine Diack as president of the IAAF in August 2015.
BBC economics editor Robert Peston had suggested that, with the support of the prime minister and chancellor, Lord Coe was a "virtual shoo-in" to succeed Lord Patten.
Previously, Lord Coe told BBC Radio 5 live he was "not sure" whether he would apply for the job.
Lord Patten stood down as chairman of the corporation's governing body in May after major heart surgery.
Vice-chairman Diane Coyle took over as acting chairman until a successor is appointed.
The BBC Trust is the guardian of licence fee revenue and of the public interest in the BBC.