The university access watchdog says students' backgrounds should be taken into account when awarding places, to improve "equality of opportunity".
A-level grades are a "robust measure" only if the applicants' "context" is also considered, Chris Millward says.
Many universities give extra help to disadvantaged applicants - but a report warns of a lack of openness about how this operates.
All Russell Group universities use some form of "contextual admissions".
Top universities have faced accusations of being socially exclusive and recruiting too few applicants from ethnic minorities.
But they also face scrutiny for being unfair to individual applicants who might lose out on places to disadvantaged candidates with worse results.
A report from the Fair Education Alliance campaign group says there needs to be much more transparency about how universities use the background of applicants when making offers and awarding places.
The campaign group, of more than 100 education and business organisations, says this can include taking into account family income, whether an applicant lives in a deprived area or if they attended a school with poor exam results or where few pupils go on to university.
There might be extra consideration given to applications from disadvantaged pupils or they might be offered places on lower grades.
But the report says there needs to be much more clarity about these decisions and how different forms of disadvantage are defined.
Research for the report, carried out by the University of Exeter, shows the extent of the challenge - with figures showing how few places in 2016 were awarded to applicants from areas with few young people going to university.
The University of Cambridge had only 3% of entrants from such "low participation neighbourhoods", the University of Bristol 3.7%, Oxford 4.6% and Exeter 5.3%.
'Where a student grew up'
Mr Millward, the Office for Students' director of fair access and participation, said: "We are a long way from equality of opportunity in relation to access to higher education."
Universities are in charge of their own admissions, so the fair access director can encourage but not instruct.
But Mr Millward said: "An ambitious approach to contextual admissions must be central to our strategy if we are going to make progress on access at the scale and pace necessary to meet the expectations of government, students and the wider public.
"A-level grades can only be considered to be a robust measure of potential if they are considered alongside the context in which they are achieved," he said.
Sarah Stevens, of the Russell Group, said that all of its universities used "contextual data" in some form.
"Qualifications and predicted grades are a key indicator of academic ability - but universities take a range of other factors into account to understand the applicant's achievements in context," she said.
"This includes the school or college attended, where a student grew up, whether they are a care leaver, or whether they are the first in their family to enter higher education."