David Cameron has accused the current prime minister, Boris Johnson, and Michael Gove of behaving "appallingly" during the EU referendum campaign.
Speaking to the Times ahead of the launch of his memoir, the former Tory PM attacked some colleagues who backed Leave for "trashing the government".
Mr Cameron said the result in 2016 had left him "hugely depressed" and he knew "some people will never forgive me".
He also said another referendum cannot be ruled out "because we're stuck".
Mr Cameron criticised Mr Johnson's strategy for dealing with Brexit, including his decision to suspend Parliament ahead of the 31 October deadline and removing the whip from 21 Tory MPs who voted to block a no-deal Brexit.
The prime minister has said the suspension - or prorogation - is a normal action of a new government to let it lay out its new policies in a Queen's Speech, and blocking no-deal would "scupper" his negotiations with the EU.
Mr Cameron called the referendum in 2016 after promising it in the Conservative Party's election manifesto the year before.
He campaigned for Remain, but lost the vote by 52% to 48%, and announced within hours he would be stepping down as PM.
The former Tory leader said the Leave side had a "very powerful emotional argument", while Remain had the "very strong technical and economic arguments", and the former - plus the issue of immigration - was a "winning combination" for his rivals.
"It turned into this terrible Tory psychodrama and I couldn't seem to get through," he said.
But leading Brexiteer and former Conservative cabinet minister Lord Lilley said the 17.4 million people who voted to leave the EU "didn't care a fig about Tory psychodramas or anything else", accusing Mr Cameron of using "an extraordinary Westminster bubble phrase".
"Most [Leave voters] put aside party loyalties and voted on the issue," he told BBC Two's Newsnight programme.
"When the British people speak, their voice will be respected, not ignored."
Lord Lilley said Mr Cameron had vowed before the 2016 referendum the public would decide whether the UK left the EU, but "now he's saying different things".
In his interview with the Times, Mr Cameron - who was prime minister between 2010 and 2016 - said his Conservative colleagues Mr Johnson, Mr Gove, Penny Mordaunt and Priti Patel had "left the truth at home" on the referendum campaign trail, especially when it came to immigration.
He said: "Boris had never argued for leaving the EU, right?
"Michael was a very strong Eurosceptic, but someone whom I'd known as this liberal, compassionate, rational Conservative ended up making arguments about Turkey [joining the EU] and [the UK] being swamped and what have you."
Mr Cameron called it "ridiculous" and "just not true" when Ms Mordaunt made a similar argument about Turkey, followed by claims by the now-Home Secretary Ms Patel that "wealthy people didn't understand the problems of immigration".
He added: "I suppose some people would say all is fair in love and war and political campaigns. I thought there were places Conservatives wouldn't go against each other. And they did."
Despite his criticism of his former colleagues' conduct during the referendum campaign, Mr Cameron defended his decision to call the vote, saying the issue of the EU "needed to be addressed".
"Every single day I think about it, the referendum and the fact that we lost and the consequences and the things that could have been done differently, and I worry desperately about what is going to happen next," he said.
"I think we can get to a situation where we leave but we are friends, neighbours and partners. We can get there, but I would love to fast-forward to that moment because it's painful for the country and it's painful to watch."
Speaking about the current prime minister's strategy, Mr Cameron said he "wants him to succeed", but his plan has "morphed into something quite different".
He said: "Taking the whip from hard-working Conservative MPs and sharp practices using prorogation of Parliament have rebounded.
"I didn't support either of those things. Neither do I think a no-deal Brexit is a good idea."
Cameron's return may play badly for Johnson
By Helen Catt, BBC political correspondent
David Cameron has been very quiet since he walked out of Downing Street for the last time in 2016.
So his decision to use this interview to come out fighting for why he called the referendum is significant.
Despite admitting that he worries about the consequences and accepting he may be blamed for them by some, he doesn't believe he was wrong to call it.
Instead, he maintains that holding the vote was "inevitable".
After years of silence, the timing of Mr Cameron's return to the front pages may play badly for Boris Johnson.
He's highly critical of Mr Johnson's role in the Leave campaign, writing in his book that he and his fellow Leave campaigner Michael Gove behaved "appallingly".
And although he seemed to be giving Mr Johnson breathing space as the new prime minister, the decision to suspend Parliament and expel 21 Conservative rebels seems to have hardened his tone.
Mr Cameron also spoke of the damage to his friendships - including the one between him and Mr Gove, who had been close friends since university.
"We've spoken," he said. "Not a huge amount. I've sort of had a conversation with him.
"I've spoken to the prime minister a little bit, mainly through texts, but Michael was a very good friend. So that has been more difficult."
But he did praise his immediate successor, Theresa May, who had been his home secretary throughout his time at No 10, for her "phenomenal" work rate and her "ethos of public service", even if he was not unquestioning of her strategy.
"I remember frequently texting [Mrs May] about the frustration of getting a Brexit deal and then seeing Brexiteers vote it down, possibly at the risk of the whole project they had devoted themselves to," said Mr Cameron. "Maddening and infuriating."
He continued: "There's an argument that Brexit is just impossible to deliver and no one could have done, and there's an argument that, well, wrong choices were made. This is somewhere in between."
Asked what happens next, Mr Cameron said he did not think a no-deal Brexit "should be pursued".
He also did not reject a further referendum.
"I don't think you can rule it out because we're stuck," he said.
"I'm not saying one will happen or should happen. I'm just saying that you can't rule things out right now because you've got to find some way of unblocking the blockage."
David Cameron as PM
Mr Cameron became the Conservative Party leader in 2005. Five years later he was voted into Downing Street as the UK's youngest prime minister in almost 200 years - aged 43.
His six-year tenure - firstly in coalition with the Liberal Democrats and latterly with a majority government - was dominated by his desire to reduce the deficit, and the introduction of austerity measures with his Chancellor George Osborne.
But when he pledged in his party's 2015 manifesto to hold a referendum on the UK's membership of the EU, the focus shifted.
Mr Cameron backed Remain during the 2016 campaign and, on the morning of the result after discovering he had lost, he announced he would be stepping down, saying: "I do not think it would be right for me to try to be the captain that steers our country to its next destination."
The former PM has remained silent until now about both of his successors at the helm of the Tory Party - Theresa May and Boris Johnson.
But his allegedly fractious relationship with Mr Johnson has been well documented since their days together at Oxford University - most notably as members of the infamous Bullingdon Club.