Labour's Ken Livingstone and the Green Party's Jenny Jones have pulled out of a BBC London mayoral debate because the BNP candidate is taking part.
A spokesman for Conservative Boris Johnson said he had never confirmed his involvement, but would also "not share a stage" with the BNP.
Lib Dem Brian Paddick has withdrawn too, but a spokesman said it was not a result of the BNP's participation.
The BBC said it was committed to giving the BNP "proportionate coverage".
A spokesman for the BNP said the withdrawals were "all rather childish" and would put people off politics: "They are just trying to scupper another person's opportunity for debate, which is very uncivilised and very un-British."
Londoners will go to the polls to elect a new mayor on 3 May.
The debate, due to be hosted by BBC London 94.9 on Monday, is the latest in a series of mayoral broadcasts but most have featured just the candidates from the biggest parties.
'Point of principle'
Announcing his withdrawal, Mr Livingstone said: "The far right want to destroy our democracy and stand for the elimination of our basic rights. They cannot be treated as a legitimate part of politics.
"We have been in negotiation about a debate with the main candidates for BBC London 94.9 - but only now have we been informed that the BNP had been invited to take part in this debate.
"I will not share a platform with the BNP and it is a point of principle to me that I never will do."
A Green Party spokesman also said Ms Jones "would not share a platform or a debate" with the BNP's candidate Carlos Cortiglia.
He said she still very much wanted to take part in the event, but could not do so unless the arrangements for it were changed.
Mr Johnson's campaign team, meanwhile, said they were offered a debate "with all four candidates" but had not yet confirmed because of an existing "time clash".
The BNP's involvement was "not discussed", a spokeswoman for the Conservative mayor said, but added: "Boris would not share a stage with the BNP."
A spokesman for Lib Dem candidate Mr Paddick said it would not be possible to have "a serious debate about the future of London" without the incumbent mayor and Mr Livingstone.
"We are not withdrawing because of the BNP's participation but we would not criticise other parties for taking this decision," he said.
"Liberal Democrats have taken the BNP head on in the past, such as when Chris Huhne went on Question Time with Nick Griffin."
Despite those withdrawals, a spokeswoman for UK Independence Party candidate Lawrence Webb said he was still planning to take part.
"We are going to take every opportunity to talk to Londoners, particularly as we are fourth in the polls but are being treated as an also-ran."
She said it was "pathetic" to pull out of the debate over the BNP adding: "Are there any toys left in their pram?"
Independent Siobhan Benita said: "For me it's just really disappointing because this was the only platform the BBC had given me alongside Boris and Ken.
"My feeling is all seven candidates have been selected, all seven should have a voice."
A spokesman for the BBC said it wanted to hear from all the candidates in a single studio debate, but that was "a big logistical challenge" and "might not be possible".
"It is up to individuals to decide whether to take part," he said.
"The BNP has demonstrated evidence of electoral support - over a series of different types of elections and by winning representation in the London Assembly in 2008 - for which we will give them proportionate coverage."
'Not the issue'
Earlier, BNP candidate Mr Cortiglia, who is originally from Uruguay, told the BBC's Daily Politics that immigrants "who are legally entitled to be in this country" were "not the issue" for the British National Party.
He said illegal immigration and bogus asylum seekers were the party's "bone of contention" - despite its 2010 general election manifesto pledge to campaign against "the immigration invasion of our country".
Mr Cortiglia also said the BNP "has to change" and "stop talking about 1930s ideas".
Mr Livingstone has reacted angrily to reports that Mr Cortiglia is planning to give him his second preference vote in the mayoral vote.
"That the BNP candidate may have called for a second preference for me is clearly designed to disorganise progressive opinion and divide opposition to them," he said.
But the BNP spokesman said the reports were not correct: "I have spoken to Carlos and he certainly hadn't said that."