Governments and employers should invest more in jobs for graduates, former university leader Prof Sir Deian Hopkin has said.
It comes as a report revealed more graduates left Wales than arrived between 2013 and 2016.
The proportion of graduates in jobs that do not require a degree was also higher than other nations or regions.
The Welsh Government said its retention of graduates was similar to many other parts of the UK.
Prof Sir Deian said graduate jobs had not been created by firms at the rate public sector jobs were lost.
Among Welsh graduates who went to university in Wales and stayed for work, 40.6% are in non-graduate jobs, the report by the Resolution Foundation think-tank said.
Scotland was in second place with 40.3%.
Wales attracted 23,807 graduates between 2013 and 2016, but 44,335 left - a difference of 20,528.
The only areas where this difference was greater, was Yorkshire and Humber (23,115) and the north-east of England (22,543).
Prof Sir Deian - a former vice-chancellor of London South Bank University - told BBC Radio Cymru's Post Cyntaf programme that graduate jobs had been lost in the public sector, but had not been created at the same rate in the private sector.
"This is a difficult circle to close, because the government should invest more and more in creating jobs that will attract and keep graduates in Wales," he said.
"But if employers are not investing at the same time there's a restriction on government - they can't do that much."
Stephen Clarke, of the Resolution Foundation, said the data did not show what kind of graduates were leaving.
"So in that case, the word 'brain drain' is perhaps appropriate and it is something that should be a concern," he said.
"One thing that is perhaps also worth saying is there are other areas of the country that see this and it might not just be purely economic - there could be geographic and social reasons as well."
Andrew Henley, an economist at Cardiff Business School, said the loss of graduates was "really quite worrying".
While an adviser to the first minister until 2012, Prof Henley said higher education officials thought the number of students leaving Wales to study "was more or less balanced" by those coming here.
"These are different data in a way," he said, referring to the Resolution Foundation report.
"But they are suggesting that over time those graduates who we are educating in Wales are not staying in Wales - they are disappearing back to England where there are perhaps better paid jobs."
Conservative shadow education secretary Darren Millar said: "It should come as no surprise that Wales is experiencing a brain drain when the Welsh Labour Government is failing to deliver the dynamic modern economy that we need."
UKIP said it would scrap tuition fees for science and technology courses at university.
Plaid Cymru would offer graduates £6,000 a year to help pay off debts if they stay or return to Wales after university.
The Welsh Government said it remained committed to making the investment Wales needs, and would develop opportunities for young people.
"We recognise that offering incentives for graduates to remain in, or return to, Wales could be part of this," a spokesman said, adding it was taking action including a new commitment for those who receive nursing bursaries to work in the Welsh NHS.