PM David Cameron has effectively vetoed an EU-wide treaty change to tackle the eurozone crisis, saying it was not in the UK's interests.
Instead a new "accord" setting out tougher budget rules will be drawn up for the eurozone, which all EU states, except the UK, look set to join.
France's Nicolas Sarkozy said the UK PM had made "unacceptable" demands.
UK Foreign Secretary William Hague denied the move would leave the UK isolated in Europe.
He said signing up to a change to the Lisbon Treaty - the treaty which governs the running of the EU - would have meant giving up more national sovereignty.
Nearly 10 hours of overnight talks could not produce an agreement involving all member states, but the 17 eurozone countries and the other EU states apart from the UK - are expected to sign up to the new deal, which includes:
- a commitment to "balanced budgets" for eurozone countries- defined as a structural deficit no greater than 0.5% of gross domestic product - to be written into national constitutions
- automatic sanctions for any eurozone country whose deficit exceeds 3% of GDP
- a requirement to submit their national budgets to the European Commission, which will have the power to request that they be revised
The UK has long resisted calls from other EU leaders for a Europe-wide tax on financial transactions - a so-called Tobin tax - which it argues would hit the City of London disproportionately.
French President Nicolas Sarkozy said Mr Cameron had made "unacceptable" demands and the sticking point had been Mr Cameron's insistence on a protocol allowing London to opt out of proposed change on financial services.
Mr Sarkozy said he would have preferred a treaty change involving all 27 EU members but "that wasn't possible, given the position of our British friends".
"We were not able to accept [the British demands] because we consider quite the contrary - that a very large and substantial amount of the problems we are facing around the world are a result of lack of regulation of financial services and therefore can't have a waiver for the United Kingdom."
But Mr Cameron told a news conference that the deal on the table was not in Britain's interest "so I didn't sign up to it".
"We want the eurozone countries to come together and solve their problems. But we should only allow that to happen within the EU treaties if there are proper protections for the single market, for other key British interests" he said.
"Without those safeguards it is better not to have a treaty within a treaty, but have those countries make their arrangements separately.
"It was a tough decision but the right one."
The BBC's political editor Nick Robinson said there was no denying now that a two-speed Europe - those inside the new deal and those outside - was inevitable.
He predicted a series of legal challenges about what the new euro "club within a club" could discuss, and whether it should be allowed to use EU resources and officials.
Foreign Secretary Mr Hague told the BBC other European countries had made "nothing like enough of an effort" to meet UK concerns.
He said the UK had sought - but not been given - guarantees that no decisions would be taken that would disadvantage those nations outside the eurozone, or disadvantage the financial services sector in the UK.
He denied that the decision left Britain isolated, because he said the new club was not one the UK would want to be part of.
"What they've committed themselves to here is to giving up more national control over their budgets, and us standing apart from that is not being isolated from them, it is a very sensible thing to stand apart from that," he said.
Deputy Prime Minister Nick Clegg - whose Liberal Democrat party is much more pro-European than their coalition colleagues - said he "regretted" that a deal involving all 27 members could not be reached.
But he added: "The demands Britain made for safeguards, on which the coalition government was united, were modest and reasonable. They were safeguards for the single market, not just the UK.
"There were no demands of repatriation of powers from the EU to Britain and no demands for a unilateral carve-out of UK financial services."
Conservative Mayor of London Boris Johnson told the BBC: "David Cameron has played a blinder and he's done the only thing that it was really open for him to do."
But Labour leader Ed Miliband said Mr Cameron had failed to build alliances before the summit.
His shadow foreign secretary Douglas Alexander said Mr Cameron had not secured any additional protections for the City of London, despite stating that that was his priority.
Nigel Farage, leader of the UK Independence Party, which campaigns for the UK's exit from the EU, said: "It's quite untenable for us to remain in a union alone, on the outside, having laws made for us, [while we're] in a permanent voting minority.
"This is the worst of all worlds for the UK."
The BBC's business editor Robert Peston said the announcements so far from Brussels had not been enough to satisfy the bond markets - as demonstrated by a rise in the implicit borrowing costs for two heavily-indebted members of the eurozone, Italy and Spain.