The 29bn euro (£24.5bn) merger of the London Stock Exchange and Deutsche Boerse could collapse after the LSE said the deal was unlikely to be approved by the European Commission.
The commission had ordered the LSE to sell its 60% stake in MTS, a fixed-income trading platform.
However, the LSE said the request was "disproportionate".
It warned investors it would struggle to sell MTS and that such a sale would harm its ongoing business.
As a result, the LSE said: "Based on the commission's current position, LSE believes that the commission is unlikely to provide clearance for the merger."
The two rival exchanges announced plans for a "merger of equals" about a year ago, aiming to create a giant trading powerhouse that would better compete against US rivals.
They had already agreed to sell part of LSE's clearing business, LCH, to satisfy competition concerns before the commission's surprise demand concerning MTS earlier this month.
The Commission had given the exchanges until Monday to come up with a proposal to meet that demand.
LSE said that such a sale would need regulatory approval from several European governments and would hurt its wider Italian business.
"Taking all relevant factors into account, and acting in the best interests of shareholders, the LSE Board today concluded that it could not commit to the divestment of MTS," the exchange said on Sunday night.
Dominic O'Connell, Today business presenter
When the London Stock Exchange trumpeted its merger with Deutsche Boerse last year, the chairman, Donald Brydon, was remarkably relaxed about the Brexit vote. He and his German counterparts said the deal would go ahead regardless of the outcome. What they didn't know was that it would eventually fall foul of another aspect of the single market - Brussels' control of competition laws.
The two sides have made no bones about the fact the merger would create a company with a powerful presence in key markets, and have offered a number of sops to competition authorities in an attempt to win their approval. But it was not enough; Brussels insisted the LSE cede control of MTS, an important Italian clearing house with a crucial role in the trading of Italian government debt.
For the LSE board, this was a step too far. A sale of MTS would be too harmful, the directors decided, and they have chosen not to meet an European deadline to say how they would get rid of it. The Brits say they want to go ahead with the merger, but it looks like an insurmountable obstacle. The LSE will either go it alone in a post-Brexit world, or - as has often seemed possible before, and even more so now - fall into the arms of one of the giant American exchanges.
MTS is a relatively small part of LSE's business, but it is a major platform for trading European government bonds, particularly in Italy, where it is classified as a "systemically important regulated business".
The LSE group also owns the Milan-based Borsa Italiana.
The LSE said it remained convinced about the merits of a merger, but joining forces with Deutsche Boerse would be impossible unless the commission changed its stance.
Deutsche Boerse said on Sunday night that it and the LSE would await a further assessment by the European Commission, which was expected to make a decision by the end of March.
Syed Kamall, MEP for London and a member of the European Parliament's financial services committee, told the BBC: "I think this is good news for competition, and it's also good news for those who are worried that some business in London might be moved to Frankfurt via the back door."
The European Commission declined to comment.
Shares in the London Stock Exchange fell 3% on the news.