Scottish independence: Report details concerns over rest of the UK tuition fees
An independent Scotland could no longer charge students from the rest of the UK, according to experts questioned in a University of Edinburgh survey.
The report also said concerns that the change would lead to an influx of students were unfounded.
The Scottish government said any increase in UK students would put places for Scottish students at risk.
Liam McArthur, Scottish Lib Dem education spokesman, said the report was a "challenge" to government plans.
Currently, students from the rest of the UK have to pay tuition fees to study in Scotland.
However, the report, in which researchers interviewed 50 higher education policy makers and "other stakeholders in Scotland", has highlighted doubts about whether this policy could continue post-independence.
The report says: "Were Scotland to become an independent EU member state, many commentators believe that they would have to be treated in the same way as EU and Scottish students, and this might make the policy of free higher education unsustainable."
However, it is generally believed concern about an influx of students from England choosing free education in Scotland to avoid paying fees at home is unfounded, according to the survey results.
The report said: "The increase in the proportion of such students coming to Scotland would be relatively small in the short term.
"As our interviews with young people indicated, the cost of study is affected by many other things than fees."
The Scottish government said any increase in students from the rest of the UK would put university places for Scottish students at risk, so it has an "objective justification" for continuing to charge English students after independence.
A Scottish government spokesman said: "Even a limited increase in the proportion of students from England could have a significant impact on access to places for Scottish-domiciled students.
"The ability to benefit from free tuition would provide a very strong incentive for students being asked to pay up to £27,000 over three years to come to Scotland in increased numbers.
"That is why our policy on this matter is in place."
However, commenting on the paper, Lib Dem Mr McArthur said: "As part of the UK, Scottish universities are at the top of the global league tables for the quality education they provide, attracting growing numbers of students from across the globe.
"Today's report from education experts presents a square challenge to the nationalists and both students and the university sector deserve answers from Mr Salmond and colleagues instead of more bluster."