The BBC has dropped plans to hold a Brexit debate between Theresa May and Jeremy Corbyn, saying it "could not reach an agreement" on its proposal.
Both it and ITV offered to air a debate between the leaders on Sunday - two days before Parliament votes on Brexit.
But Labour had said the BBC's proposed format was a "mish-mash, with a lopsided panel of other politicians and public figures" taking part.
No 10 said Labour's objections were "false and flimsy".
The BBC had wanted to include "a range of voices" in the programme, including members of other political parties, as well as a head-to-head between the leaders.
The Downing Street spokesman said: "We remain committed to holding a debate and will continue to press for a format that ensures a range of voices are heard alongside a substantial head to head."
Several other parties had said they should be included in the debate - including the SNP, Lib Dems, Plaid Cymru and Greens.
ITV says "invitations remain open" to both leaders to hold a debate on the channel this Sunday.
The news about the BBC debate came as MPs geared up to begin five days of debate in Parliament over the Brexit withdrawal agreement.
What have the channels said?
The BBC said last Thursday that the prime minister had accepted its offer to take part in a debate on Sunday night and it was waiting to hear from the Labour Party.
Mr Corbyn then told ITV's This Morning that he preferred ITV's offer - partly because it would not clash with the final of I'm A Celebrity... Get Me Out Of Here! on the same evening.
On Saturday, the BBC released a further statement, saying its proposal included both a head-to-head debate and "an opportunity to hear from a wider range of voices".
And in a statement on Monday night, ITV said "invitations remained open" to both leaders for its straight head-to-head debate - it confirmed this was still the case after the BBC said it was pulling out.
In a statement on Tuesday, the BBC said its proposed format had included:
- A head-to-head debate between the leaders
- A discussion between eight panellists with a wide range of views on Brexit - including other politicians
- A further head-to-head debate
- Closing statements from each leader
"We have been clear throughout the whole of this process that, as well as a substantive head-to-head debate, any programme we broadcast would need to include other voices, including other political parties, to reflect the wide range of views the public and parliamentarians hold about Brexit," it said.
"We believe ours was a fair and appropriate format for those taking part and, crucially, for our audiences around the country, and it is a shame we will not be able to bring them this programme."
Earlier on Tuesday, Labour criticised Mrs May and her team for choosing their "preferred broadcaster" for the debate.
The party's spokesman said she "running away from the scrutiny" of a head-to-head with Mr Corbyn by accepting the format over ITV's.
But a Downing Street spokesman in turn accused Mr Corbyn of "running scared of proper scrutiny".
"The prime minister issued the original challenge to Jeremy Corbyn for a head-to-head debate, and despite her flexibility on timing and format, Jeremy Corbyn and Labour have done nothing but raise false and flimsy objections to the BBC's proposal."