Scottish banks might not be bailed out in a financial crisis if an independent Scotland had no formal currency union with the rest of the UK, a report says.
The SNP says Scotland would keep the pound in a formal union - an option ruled out by the three main UK parties.
The National Institute of Social and Economic Research said without a formal arrangement Scottish banks would need a new "lender of last resort".
Labour's Ed Miliband has urged the SNP to set out a currency "Plan B".
It comes as the question of which currency an independent Scotland would adopt has become an increasingly key issue in the referendum campaign.
First Minister Alex Salmond has said a formal currency union with the rest of the UK - in which an independent Scotland would keep the pound - was the best option, saying: "It's Scotland's pound and we are keeping it."
He has argued that a currency union would be in the interests of Scotland and the rest of the UK.
But Labour, Conservative and Liberal Democrat leaders have all ruled out a currency union with an independent Scotland and have urged Mr Salmond to come up with a currency "Plan B"
The National Institute of Social and Economic Research considered other currency options, including "sterlingisation" - in which Scotland continued to use the pound without a formal agreement with the rest of the UK and without the Bank of England as lender of last resort.
It warned options for creating a lender of last resort for Scottish institutions may involve "terms that are unlikely to be acceptable to an independent government".
The report also said an informal union would have consequences for Scotland's financial sector and its capacity to export financial services.
Analysis by Colletta Smith, BBC Scotland Economics Correspondent
This report tackles an issue that neither side of the referendum campaign are prepared to discuss openly.
With a political impasse being reached over an idea of a formal currency union, academics are now turning their attention to the viability of other options mentioned in the White Paper.
In a very strongly-worded report, the National Institute of Economic and Social Research makes it clear that an informal currency union would have an incredibly high economic cost.
Without the backing of the Bank of England and with no viable alternative as a lender of last resort, big financial companies may have no choice but to leave Scotland.
RBS, Lloyds, and Standard Life have already warned that they would try and minimise any risk to their institutions. The NIESR report suggests that without a currency union, having big UK banks outside the legal jurisdiction would not be acceptable for the UK regulator.
Chief Secretary to the Treasury Danny Alexander said the report confirmed that "borrowing the pound would be damaging for a separate Scotland's economy and a terrible choice for Scotland's financial sector".
"This report shows that the Plan B Salmond now seems to be flirting with would be disastrous for the Scottish economy," he said.
But a Scottish Government spokeswoman said the pro-independence campaign's position was to keep the pound.
"In fact this report demonstrates precisely why a currency union and a shared system of financial stability is in the overwhelming interest of Scotland and the rest of the UK," the spokeswoman said.