David Miliband has said he "feared being a distraction" and hopes his decision to leave UK politics helps his brother Ed to lead Labour against the coalition in an "uninhibited way".
The former foreign secretary, pipped to Labour's leadership by his younger brother in 2010, is to step down as an MP and join a US-based charity.
He told the BBC it had been a "very difficult" decision to make.
Ed Miliband said the move would leave British politics "a poorer place".
But after serving as an MP for 12 years, David Miliband said: "I now have to make a choice about how to give full vent to my ideas and ideals."
In an interview with BBC political editor Nick Robinson, he said: "I feared being a distraction in whatever role I played at Westminster.
"I feel a sense of sadness because I am British. I love Britain. I am passionate about Labour, but I have had to make a choice about where I can make my best contribution."
He said it had been "hard for me to accept that I can best help the Labour Party by not just giving the space between the front bench and the back bench to Ed, but the space between the front bench and 3,000 miles away".
"I have wrestled with this very, very hard, and I have tried to make a decision that I honestly say to Labour members and supporters that is right for me and for the Labour Party.
"I want it to be the vision Ed Miliband has versus the vision David Cameron has. Not Ed and David Miliband. I didn't want to become a distraction. I didn't want the soap opera to take over the real substance of what needs to be done."
The former foreign secretary did not rule out an eventual come back in British politics, but said: "I am taking a job in America, not taking citizenship in America. I will continue to follow what's going on here, but my focus is going to be making a difference through the International Rescue Committee."
'Time has helped heal it'
David Miliband was long seen as a future Labour leader, with supporters of Tony Blair pushing him to stand against Gordon Brown when Mr Blair stepped down as prime minister and Labour leader in 2007.
There were also frequent reports that he was set to challenge Mr Brown during the three years he was prime minister before leading Labour to defeat at the 2010 election.
Instead of challenging, Mr Miliband bided his time and entered the post-election Labour leadership contest as overwhelming favourite - only to lose to his brother, who gained more union votes but fewer votes from Labour members and MPs.
The disappointment and strained relations led to David Miliband deciding to step down from the Labour front bench to, as he put it in his letter of resignation, "give Ed the space and the same time the support he needed to lead the party without distraction".
Wednesday's announcement seemingly brings to an end the almost constant rumours that he was set for a return to the opposition front bench.
In his statement Ed Miliband said: "Having spoken to him a lot over the past few months, I know how long and hard he thought about this before deciding to take up the offer. I also know how enthusiastic he is about the potential this job provides.
"We went through a difficult leadership contest but time has helped to heal that. I will miss him. But although he is moving to America, I know he will always be there to offer support and advice when I need it."
David Miliband's decision will spark a by-election in South Shields, where he has been MP since 2001, although the timing of any vote is not yet known.
Nick Robinson said Mr Miliband had "clearly concluded he does not want to return to the fray, he doesn't want to serve under his brother in opposition or in government which is a serious blow to his brother, and disappointment to members of the party".
In his letter to his constituency party chairman, Mr Miliband said the International Rescue Committee was founded in the 1930s at Albert Einstein's suggestion to help people fleeing the Nazis. And his own family history - his parents both fled Germany in the 1930s - meant "I feel that in doing this job I will be repaying a personal debt".
"This job brings together my personal story and political life. It represents a new challenge and a new start," he said.
The MP, who is vice-chairman and non-executive director of Sunderland Football Club, is married to violinist Louise Shackelton - who has dual UK/US citizenship - and the couple have two children.
Tony Blair, former Labour leader and prime minister, said: "I congratulate David on his appointment to a major international position. It shows the huge regard in which David is held worldwide. I'm sure he will do a great job. He is obviously a massive loss to UK politics.
"He was the head of my policy unit and then a truly distinguished minister in the government and remains one of the most capable progressive thinkers and leaders globally. I hope and believe this is time out, not time over."
David Miliband's former cabinet colleagues, Lord Mandelson and Jack Straw, said they did not think it was the end of his political career.
"I think he has a future in politics... I think I know a little bit about comebacks in politics and, to coin a phrase, if I can come back [then] David Miliband can come back - and I think he will," said Lord Mandelson.
Mr Straw said he would be "welcomed back into the Labour movement".
Conservative Party chairman Grant Shapps added: "He has contributed a great deal to British politics and we wish him well."
Former US President Bill Clinton congratulated the charity on appointing Mr Miliband, saying: "I have known David almost 20 years. He is one of the ablest, most creative public servants of our time."
But Labour MP John Mann described Mr Miliband as "the man who would have been prime minister if he had ever asked 'And what is your view'?"