Some of Gordon Brown's senior cabinet ministers took part in the election campaign believing Labour could not win under him, Lord Mandelson has claimed.
The ex-business secretary, in his memoirs, says the then PM feared voters did not want "five more years" of him.
He recalls minister Douglas Alexander calling Labour's position "futile", while Chancellor Alistair Darling said the party was "going to lose".
The PM offered to resign within a year if he won the election, the book adds.
The Times, which has paid to serialise extracts from Lord Mandelson's book The Third Man, also suggests there were disputes between 10 Downing Street and the Treasury over tax policy.
The book, due to be published later this month, is the latest in a series of reminiscences by key figures in New Labour.
Former No 10 communications director Alastair Campbell published the first uncut volume of his diaries last month while former Prime Minister Tony Blair's autobiography will go on sale in September. Mr Brown, who stood down after Labour lost the election in May, is also writing a book.
Publishing a second day of extracts from Lord Mandelson's memoirs, the Times says he quoted a string of ministers casting doubt on whether a fourth-term Labour victory was possible under Mr Brown.
It says he recounted a cabinet meeting last October, in which deputy leader Harriet Harman urged Labour's campaign to focus on future, family and fairness.
Lord Mandelson, who was chairing the meeting, adds that he, Mr Alexander and Mr Darling instead proposed three "F-words" of their own: "futile"; "finished"; and "f***ed".
He suggests Mr Darling told him before the end of 2009 that Labour was "going to lose" while Lord Mandelson himself acknowledged the same to then Foreign Secretary David Miliband.
As well as offering to stay in office for a year - if Labour had won - to oversee the recovery and then make way, Lord Mandelson also states Mr Brown offered to hold a "mega-referendum" on a number of issues in 2011 and to quit if this did not go his way.
In an interview accompanying the serialisation, Lord Mandelson told the Times that there was a "raw" and "agonised" debate about the state of the party and its leadership in the summer and autumn of 2009.
However, he said he did not need to "think twice" that Mr Brown should stay in place amid attempts to unseat him by colleagues in July.
Removing Mr Brown then - given the state of the economy and the major decisions that he had taken - would not have been putting the country's "interest first".
Amid reported wrangling between No 10 and the Treasury, Lord Mandelson suggested in his memoir that Mr Brown rejected a proposal from the chancellor to raise VAT while Mr Darling quashed calls for any future VAT rises to be ruled out.
The former business secretary also claims that, towards the end of Mr Blair's time in power, there was an "insurgency" among Mr Brown's supporters, pushing for him to become prime minister.
Questioned on BBC Two's Daily Politics about his alleged comments, Mr Darling said: "It was always obvious for anyone to see that we had real difficulties."
But he refused to provide a "running commentary", adding that there would "be a time when a mature consideration of what Labour did in its 13 years, the good as well as the not so good", was appropriate.
Speaking on BBC Radio 4's Today programme, shadow education secretary and Labour leadership contender Ed Balls - a close Brown ally - dismissed suggestions that he had been part of an attempted "coup" against Tony Blair as "absolute total nonsense".
He said: "I was never involved in an insurgency. I was very close to Gordon Brown but I also saw Tony Blair very regularly. But we had our disagreements."
Lord Mandelson was brought back into the cabinet by Gordon Brown in 2008, having previously been thought to have been at loggerheads with him.
The peer, who resigned twice from the government under Tony Blair, stood down from Labour's front bench after its election defeat.