Scots Tory leader Ruth Davidson and Scottish Secretary David Mundell could resign from their roles over a possible Brexit compromise, BBC sources say.
In a letter to the prime minister they said they would not support any deal that introduces different arrangements for Northern Ireland.
They said this would "undermine the integrity" of the UK.
Scotland's Brexit Minister Mike Russell said it was "an astonishing development".
Sources have told the BBC Ms Davidson and Mr Mundell have made it clear they would resign if Northern Ireland faces new controls that separate it from the rest of the UK - because that could fuel the case for Scottish independence.
A source close to Mr Mundell told the BBC: "If you find yourself not agreeing with government policy" resigning would be the "logical outcome". A source close to Ms Davidson said the issue was a "red line".
In the letter to Theresa May, the Scottish Tory leader and Scottish Secretary said: "Having fought just four years ago to keep our country together, the integrity of our United Kingdom remains the single most important issue for us in these negotiations.
'Any deal that delivers a differentiated settlement for Northern Ireland beyond the differences that already exist on all Ireland basis (eg agriculture), or can be brought under the provisions of the Belfast Agreement, would undermine the integrity of our UK internal market and this United Kingdom.
It added: "We could not support any deal that creates a border of any kind in the Irish Sea and undermines the Union or leads to Northern Ireland having a different relationship with the EU than the rest of the UK, beyond what currently exists."
‘We’ll resign if you do anything that makes Scotland think there’s a better alternative to a hard Tory Brexit’ say Mundell and Davidson! First, what a bizarre position - isn’t it their job to stand up for Scotland? And, second, have they the gumption to resign? I bet they don’t! https://t.co/mR64ApibUr— Nicola Sturgeon (@NicolaSturgeon) October 14, 2018
The Scottish government's Constitutional Relations Secretary Mike Russell told the BBC's Sunday Politics Scotland programme: "This is an astonishing development, not that Ruth Davidson holds any position in the UK government, but to say that you'll resign if Scotland is not treated as badly as everywhere else strikes me as a complete dereliction of duty.
"They're elected by people in Scotland and here they want Scotland to suffer the hardest of Brexits. That strikes me as a ludicrous position and one in which we should be saying, have they really come to this?"
First Minister Nicola Sturgeon tweeted: "We'll resign if you do anything that makes Scotland think there's a better alternative to a hard Tory Brexit' say Mundell and Davidson!
"First, what a bizarre position - isn't it their job to stand up for Scotland? And, second, have they the gumption to resign? I bet they don't!"
Labour MP Ian Murray tweeted: "But they also don't want to continue to participate in the customs union and single market so how do they suggest resolving it? Utterly contemptible.
"They can't achieve a no hard border in Ireland without participation in the same trading arrangements as the EU."