Ian Austin has become the ninth MP to quit Labour this week, blaming leader Jeremy Corbyn for "creating a culture of extremism and intolerance".
He told the BBC the leadership had failed to tackle anti-Semitism and had turned the party into a "narrow sect".
But the MP for Dudley North said he had no plans to join the new Independent Group of former Labour and Tory MPs.
Mr Corbyn denied claims bullying was rife in Labour, telling Sky News any "bad behaviour" had been dealt with.
Meanwhile, his deputy Tom Watson said Mr Austin's departure was a "serious blow" to Labour.
A Labour spokesman suggested Mr Austin should stand down and call a by-election in his West Midlands seat, which he won by only 22 votes in 2017.
Mr Austin told the Express & Star newspaper, which first broke news of his resignation, that it was the "most difficult decision" he had ever had to take.
The MP, who has represented the West Midlands constituency since 2005, later told BBC West Midlands that he was "ashamed" of the party.
"I grew up listening to my dad, who was a refugee from the Holocaust, teaching me about the evils of hatred and prejudice," he said.
"One of the main reasons I joined the Labour Party as a teenager here in Dudley more than 35 years ago was to fight racism and I could never have believed I would be leaving the Labour party because of racism too."
Analysis: Push and pull for Labour MPs malcontents
By political correspondent Jonathan Blake
Another MP resigns, pouring scathing criticism on their leader at the end of a watershed week at Westminster.
Ian Austin's decision not to join his former colleagues in the new Independent Group is telling.
It shows that he felt strongly enough about the problem of anti-Semitism within Labour to quit the party he has been a member of for 45 years on that basis alone.
But it also suggests that Parliament's newest group may be seen above all for what, in the absence of any policies, unites them.
That is support for a further referendum on leaving the EU, something Ian Austin would not sign up to.
For him the push from a party in which he no longer felt at home was more powerful that the pull of life in a new party on the outside.
Other Labour and Conservative MPs who are considering their future will also need to decide whether they're better off out than in, and whether the Independent Group is the place for them.
Explaining his decision not to call a by-election, Mr Austin said he had been openly critical of Mr Corbyn during the 2017 election campaign.
He said his "work for the people in this community is going to carry on as it always has".
Tom Watson, Labour's deputy leader, said he was "deeply saddened" by his close friend's decision to leave.
"I didn't want him to go, not just because he is a friend but because Labour needs people of his experience, calibre and passion if we are to win," he added.
But Mr Corbyn, who has been in Madrid for a meeting of European socialists, took issue with Mr Watson's comments and said he would talk to him about them in the "near future".
The Labour leader told Sky News: "There is no place for harshness, bullying or anything else in the party. I don't believe that it exists on a wide scale.
"Where there is bad behaviour we deal with it. Where there is a problem we deal with it."
Derby North MP Chris Williamson, a close ally of Mr Corbyn, said his departure was "no loss".
Describing Mr Austin as "stuck in the 1990s... a New Labour relic", he told the BBC: "I for many years wasn't particularly a fan of Tony Blair, but I didn't throw my toys out of the pram."
Mr Austin "fought the last election under false pretences", using Labour's brand to get elected, Mr Williamson added.
In an interview with London's Evening Standard, conducted before Mr Austin's resignation, shadow chancellor John McDonnell warned people behaving in a way "construed as anti-Semitic" that: "They will be dealt with. Full stop. They are not welcome."
He admitted the leadership let down Jewish MP Luciana Berger, who described the party as "institutionally anti-Semitic" when leaving to join the Independent Group.
There was not enough support for Ms Berger, "not enough action", Mr McDonnell said, before pledging to "sort it".
Mr Austin was a minister for regional affairs under Gordon Brown and part of Ed Miliband's front-bench team in opposition.
He said he "agreed" with the eight MPs who left Labour to form the Independent Group earlier this week that things "have got to change".
However, he wants a Brexit deal concluded, rather than a further referendum on EU membership.
While Mr Austin did not rule out joining the group, he told BBC Radio 5 Live he was "not anticipating doing that any time soon".
Ms Berger tweeted that she fully understood why he had come to "this difficult and painful decision" of leaving Labour.
Another of the defectors, Chuka Umunna, tweeted his "massive respect" to Mr Austin, adding: "It's painful and hard but he has stayed true to his values and what he believes to be the national interest."