Labour peers want to investigate claims senior figures in the party interfered in the disciplinary process of dealing with anti-Semitism complaints.
They said accounts of staff in a BBC Panorama programme were "powerful and shaming", but the party's response had been "heartbreaking".
They said it was now time to "rebuild confidence" in Labour's processes.
BBC political correspondent Iain Watson said shadow cabinet will meet on Monday to discuss the issue of anti-Semitism.
It was announced at a meeting of Labour MPs, where further concerns about the issue were raised - although leader Jeremy Corbyn did not attend.
Our correspondent said there was strong support at the parliamentary party for those former members of Labour's staff who had spoken to the BBC about their concerns.
The letter from the peers comes as Labour staff also condemned the party's response in a motion to their union.
When the documentary aired, a Labour spokesman said the former employees who had talked to the BBC were "disaffected", and included some officials "who have always opposed Jeremy Corbyn's leadership".
But in a motion to the GMB, staff said whistleblowers should "be commended and supported, never attacked", and called for the party to apologise to them.
Over 200 current and former staff also wrote to Mr Corbyn to say the party had treated whistleblowers in an "appalling and hypocritical" way, and that the "moral responsibility" for the anti-Semitism crisis lay with Mr Corbyn.
The Labour leader visited the party's headquarters in London on Monday to talk to staff - but did not give a full speech.
One of his local councillors in the Borough of Islington has also resigned from the party over the row.
Gary Poole, who represents the St Mary's Ward in Mr Corbyn's constituency, said he "cannot in good conscience remain a member of a political party that has singularly failed to adequately address the scourge of anti-Semitism".
The four peers who signed the letter to Mr Corbyn - Baroness Angela Smith, Lord Toby Harris, Baroness Dianne Hayter and Lord Tommy McAvoy - wrote it after three other peers resigned from the Labour Party because of anti-Semitism claims.
Lord Triesman, general secretary of the party between 2001 and 2004, accused Mr Corbyn of anti-Semitism and said the party was no longer "a safe environment" for Jewish people.
The four peers said it was "deeply saddening, but not surprising" that their colleagues had resigned due to the "toxic and endemic problem [the party] has failed to eradicate".
They also criticised the use of non-disclosure agreements (NDAs) by the party so former staff would not speak about their experiences - especially as the party had openly campaigned against them in other situations.
They said a minimum of five steps needed to take place internally:
- Ensuring a completely independent complaints process
- Automatic exclusion, similar to that used for when party members campaign against Labour
- Take the Panorama allegations seriously
- Overhaul governance arrangements of party
- End the use of NDAs
They wrote: "The purpose of these proposals is to ensure the Labour Party can regain the trust of its members, supporters and the wider public.
"Without full openness, this is a cancer that will continue to grow - and, in hurting us, it will most hurt those that need a Labour government."
Our correspondent said the BBC has been told there will be a concerted push to make the system more independent at a meeting of Labour's ruling national executive next week.