MPs have backed seeking "alternative arrangements" to replace the Irish backstop in Theresa May's Brexit plan.
The proposal - put forward by Tory MP Sir Graham Brady - had the support of the government and won by 16 votes.
Theresa May had urged MPs to vote in favour of it, to give her a mandate to return to Brussels and re-open negotiations in order to secure a "legally binding change".
But the EU has said it will not change the legal text agreed with the UK PM.
MPs voted on a string of amendments to Mrs May's plan to change the direction of Brexit.
Mrs May said that, after taking the votes into account and talking to the EU, her revised deal would be brought back to the Commons "as soon as possible" for a second "meaningful vote".
Another amendment, rejecting a no-deal Brexit, also won the support of Parliament on Tuesday, but the vote was not binding - meaning the date for exit remains 29 March.
Labour leader Jeremy Corbyn said as a result of the message from MPs rejecting no deal, he would now meet the prime minister to discuss the next steps.
He had previously refused to meet her unless she ruled out a no-deal Brexit herself.
Mr Corbyn said: "After months of refusing to take the chaos of no deal off the table, the prime minister must now face the reality that no deal is not an option."
Five other amendments, including Labour MP Yvette Cooper's bid to delay Brexit if Mrs May does not get her deal through Parliament, were defeated.
Tory MP Nick Boles, who worked with her on the amendment, tweeted out a joint statement, saying they "remain deeply concerned that there is no safeguard in place" to stop a no deal and said Mrs May's revised plan would have to "reflect the Commons opposition to no deal".
Mrs May hopes the support for Sir Graham's amendment - which won by 317 votes to 301 - to look at alternatives to the backstop gives her a stronger negotiating position with the EU.
The controversial element of Mrs May's original plan is the insurance policy to prevent checks on goods and people returning to the Northern Ireland border.
It would effectively keep the UK inside the EU's customs union, but with Northern Ireland also conforming to some rules of the single market.
It was one of the main reasons her Brexit deal was voted down in Parliament by an historic margin earlier in January as critics say a different status for Northern Ireland could threaten the existence of the United Kingdom and fear that the backstop could become permanent.
She told the Commons there was now a "substantial and sustainable" majority of MPs supporting leaving the EU with a deal, but admitted renegotiation "will not be easy".
The leader of the Democratic Unionist Party in Westminster, Nigel Dodds, said it was a "significant night" and his MPs would work with the prime minister "to deliver the right deal for the United Kingdom".
But the leader of the SNP in Westminster, Ian Blackford, said that passing the amendment had seen the government "rip up the Good Friday Agreement" - integral to the peace process in Northern Ireland.
Liberal Democrat leader Sir Vince Cable said the Commons had given the prime minister "contradictory instructions to have no deal but pursue a course of action that will lead to a no deal".
May scores 'unconventional win'
Theresa May was heading for another defeat, but she ended up with an unconventional win - a win nonetheless.
The Tory Party that was visibly split in two a fortnight ago is giving the impression of being largely united, even if that is temporary.
Yet the prime minister only won because she gave into Brexiteer and DUP demands, by making a promise that she can't be sure she can keep - one the EU says at the moment is impossible.
This process has for a long time been about No 10 stumbling, often seriously, then getting up again to try to take another step.
There is a valid question - to what end?
Speaking after the result, President of the European Council Donald Tusk said the withdrawal deal was "not open for re-negotiation" and "remains the best and only way to ensure an orderly withdrawal of the United Kingdom from the European Union".
But he said the EU would be willing to look at the political declaration again - the part of the deal that makes a pledge on the future relationship between the UK and the EU - and that the EU would "stand ready" to consider any "reasoned request" for an extension to the leave date of 29 March.
The Brexit Coordinator for the European Parliament, Guy Verhofstadt, welcomed Parliament's rejection of a no deal, but also said there was no majority in the EU to re-open or dilute the withdrawal agreement .
Welcome the UK Parliament's decision to reject a no-deal & the hope of cross-party talks on future relationship. We stand by Ireland & the Good Friday Agreement. There is no majority to re-open or dilute the Withdrawal Agreement in the @Europarl_EN including the backstop.— Guy Verhofstadt (@guyverhofstadt) January 29, 2019
A statement from the Irish government said the withdrawal agreement "is not open for renegotiation... The best way to ensure an orderly withdrawal is to ratify this agreement," the statement said.
And Austrian Chancellor Sebastian Kurz also ruled out further negotiations on the withdrawal agreement.
What happens next?
The prime minister has invited Tory MP Caroline Spelman, Labour MP Jack Dromey and others who tabled amendments to prevent a no deal to discuss how to move forward and secure a deal for Brexit.
She also invited Mr Corbyn for talks and promised the government would "redouble its efforts to get a deal this House can support".
The so-called Brady amendment could pave the way for a plan known as the "Malthouse compromise" as an alternative to the backstop.
Engineered by both Leavers and Remainers - and led by Tory minister Kit Malthouse - the proposal includes extending the transition period for a year and protecting EU citizens' rights, instead of using the backstop.
The deputy chairman of the pro-Leave European Research Group, Tory MP Steve Baker, said he hoped by the group giving its support to the amendment, MPs could "now make rapid progress towards the Malthouse compromise".
But fellow ERG member Mark Francois warned there was no guarantee the group would back the PM, and said he would wait to see what she comes back with from Brussels.
Mrs May's spokesman said she would "engage" with colleagues proposing the compromise, but would also look at other options - including putting a time limit on the backstop and seeking a way to exit it.
The PM's revised deal will return to the Commons to be voted on.
But, if it is again rejected, the government will table an amendable motion - meaning MPs can put forward more amendments as they did earlier - for debate the following day.
And if no new deal is agreed by Parliament by 13 February, she will make a statement and, again, table an amendable motion for debate the next day.