A financial body said more detail was needed on how taxes would be raised after a "Yes" vote, adding that a new system would cost hundreds of millions.
The Institute of Chartered Accountants Scotland (ICAS) said the Scottish government's independence blueprint contained "very little detail on tax".
But Holyrood ministers have said independence would create an opportunity for a simpler tax system.
People in Scotland go to the polls in September to vote in the referendum.
They will be asked the Yes/No question: "Should Scotland be an independent country?"
In its analysis, ICAS said "less complex" changes to the tax system in New Zealand cost about £750m and an independent Scotland could expect the outlay to be "significantly greater".
It also questioned the Scottish government's claims in its White Paper that set-up and running costs for a new tax system would be "a small proportion of an independent Scotland's total budget".
The report titled Scotland's Tax Future; Taxes Explained said: "No further explanation is in the White Paper, meaning it falls short of the informative and detailed financial memorandum that might be expected to accompany even the smallest piece of parliamentary legislation at Holyrood or Westminster at the moment.
"By illustration, 'small' might be 1%, which would be around £650m, but 5% would be £3.25bn."
Scotland's Finance Secretary John Swinney said the ICAS analysis failed to take account of the "long term opportunities to create a modern, efficient and cost effective tax system in an independent Scotland".
He added: "The Fiscal Commission Working Group and Institute for Fiscal Studies have both recognised the potential for Scotland to design a simpler and more efficient tax system following independence which could both save money and increase the amount of revenue collected by £250m.
"The UK has one of the most complex tax systems in the world and one of the most expensive systems in Europe.
"Independence will guarantee for the first time that decisions about taxes apply to an independent Scotland, and at what level, will only be taken with the approval of a parliament elected by people in Scotland."
ICAS has written a series of papers on financial issues linked to the referendum debate, including pensions.
Following this latest publication, the organisation's director of taxation, Elspeth Orcharton, said: "What it is really going to cost and how it is to be paid for is the question that still needs to be answered."
ICAS, which said it was "apolitical" and would "not take a stand for or against" independence, concluded that the transition costs - the one-off costs to set up a new tax system in Scotland - had yet to be fully explored or explained.
Its report examined the potential for redesigning Scotland's tax system, taking into account that income tax was presently the UK's "single biggest source of tax revenue" followed by VAT and national insurance contributions.
It said that income tax accounted for a lower percentage of total tax receipts in Scotland with the top 1% of UK taxpayers contributing 25% of income tax compared with the top 1% of Scotland's taxpayers contributing 17%.
ICAS assessed there was little scope for raising significant sums of money from increasing the 50% rate of tax, which applies to 13,000 Scottish taxpayers.
It calculated that "another 10% on the top rate" might raise about £240m or less than 0.4% of public spending, even assuming higher-rate taxpayers do not leave Scotland or find others ways to avoid paying more tax.
A Treasury spokesman said that the report highlighted the "hidden charges" that would come with settling up a tax and benefits system if Scotland voted to leave the UK.
He added: "Of course Scotland is already part of a strong and stable tax and benefits system as part of the UK and there is no need to change this."