A university group says that the government's review of tuition fees in England should make a priority of finding ways to attract more mature and part-time students.
The Million Plus group says there is a "huge pool of untapped potential" among adults who missed out on university.
After fees increased in 2012, mature student numbers fell by 20%.
Les Ebdon, head of the university access watchdog, backed calls to reverse this "very worrying trend".
Mature students - counted as people starting courses at the age of 21 or over - were among the most likely to be deterred by the raising of tuition fees to £9,000 in 2012, which have since risen again to £9,250.
Part-time students also saw a significant drop in numbers - and this often overlaps with older students, who might be working and unable to study full-time.
A report from the Million Plus group of universities says that for some courses, such as nursing, part-time, mature student numbers have fallen by 49% since fees were increased.
The Prime Minister, Theresa May, has launched a review of post-18 education, which will examine fees and financial support for students.
Million Plus, representing new universities, says there should be targeted efforts to attract more mature students, such as restoring maintenance grants, offering better financial support and more rapid write-offs for unpaid student debt.
The report, Forgotten Learners, also calls for more flexible ways of studying that would make it more practical for part-time, adult students.
The particular needs of mature learners should be recognised, says the report, such as having family responsibilities, financial pressures that make full-time study unlikely and a preference for living at home rather than on campus.
"The steep decline in mature student numbers at our universities has been tolerated for too long," said London South Bank University vice-chancellor David Phoenix, who chairs the Million Plus group.
"The UK faces a daunting productivity challenge in the decade to come and in response we need a better and more flexible offer to those over 21 to ensure that we are able to up-skill and re-skill many more people," said Prof Phoenix.
Prof Ebdon, director of the Office for Fair Access to Higher Education, said: "There is a clear societal and economic benefit to people succeeding in higher education, whatever stage of their life they come to it.
"But too often, talented people are excluded because of a lack of options that meet their needs."
Labour's shadow higher education minister, Gordon Marsden, said the "severe drop" in part-time numbers showed the government's "neglect of those who do not follow traditional routes into higher education".