First Minister Nicola Sturgeon has warned Theresa May she is not "bluffing" over her promise to hold an independence referendum if Scotland's vote against Brexit is "not respected".
After talks in Downing Street, she said she was not prepared to see "Scotland driven off a hard Brexit cliff-edge".
She and the first ministers of Wales and Northern Ireland demanded to be fully consulted on EU negotiations.
But Downing Street said it was vital not to "undermine" the UK's position.
The Scottish and Welsh leaders want the devolved legislatures and Parliament all to vote on the UK's approach when formal talks with the European Union about Brexit begin.
The issue was discussed as the joint ministerial committee - bringing together the prime minister, Ms Sturgeon, Welsh First Minister Carwyn Jones and Northern Ireland's First Minister Arlene Foster and Deputy First Minister Martin McGuinness - met for the first time since 2014.
Afterwards, Mrs May said: "Working together, the nations of the United Kingdom will make a success of leaving the European Union - and we will further strengthen our own unique and enduring union as we do so.
"The great union between us has been the cornerstone of our prosperity in the past - and it is absolutely vital to our success in the future.
"The country is facing a negotiation of tremendous importance and it is imperative that the devolved administrations play their part in making it work."
Downing Street is proposing that Brexit Secretary David Davis chair a new forum, bringing together representatives from the devolved nations before the prime minister triggers Article 50 of the Lisbon Treaty - formally starting the two-year Brexit process - by the end of March next year.
But Mr Jones and Ms Sturgeon have both said Article 50 should not be invoked until there is an agreed approach to negotiations with the EU.
The two leaders have stressed their desire to see continued participation in the European single market - a situation that could be at risk if the UK pursued a so-called "hard" approach to Brexit.
Ms Sturgeon appeared less happy than the prime minister about the way the Downing Street meeting had gone, saying she had received "no more information or detail" about the UK negotiating position, hearing only "warm words" from the UK government.
She added: "I'm not going to stand by and watch Scotland driven off a hard Brexit cliff-edge. Scotland's vote to stay has to be respected."
In 2014, Scottish voters decided by 55.3% to 44.7% against independence from the UK and Ms Sturgeon has raised the possibility of another vote.
She said: "There's not a bit of what I'm doing just now that's bluffing or game-playing. This is not a game of chicken. It's not a game at all."
Ms Sturgeon told reporters: "If all we get from the negotiations is a door slammed in our face, Scotland should be able to choose a better option."
Ms Foster, also speaking in Downing Street after the meeting, said: "It's important that we are involved in the heart of [the Brexit negotiation] process."
BBC assistant political editor Norman Smith said Mrs May had "delivered a fairly bold 'Non'" to demands for devolved legislatures to have a greater say in Brexit. She did not want "separate deals in separate parts of the UK" because this risked "confusing" issues and "undermining her negotiating strategy", he added.
But, following Ms Sturgeon's call for a second referendum, Mrs May realised what an "awfully big" situation the UK faced, Norman Smith said, adding: "If she gets it wrong, she risks being the prime minister who took Britain out of the EU but also risks breaking up the United Kingdom."
The talks came as a think tank warned the UK faced a "full-blown constitutional crisis" unless all nations of the UK agreed on "core planks" of the government's approach to Brexit.
The Institute for Government warned imposing a settlement on Scotland, Wales and Northern Ireland might be legally possible but "if it proves impossible to find consensus... the result may be a serious breakdown in relations between the four governments (and nations) of the UK".