Universities are being accused of "complacency" over a lack of senior black academics and lower achievement among ethnic minority students.
Baroness Amos, the UK's first black woman university head, says there are "deep-seated prejudices and stereotypes which need to be overcome".
"Not even 1% of UK professors are black," she will tell a conference on university leadership.
The Office for Students says tackling such "gaps" should be a priority.
Speaking ahead of the conference, Baroness Amos, director of the School of Oriental and African Studies (Soas), in London, said universities liked to see themselves as "inclusive and internationalist".
But she said there has been "anger and frustration" among ethnic minority academic staff at their under-representation in senior jobs and the achievement gap for ethnic minority students.
She will address the Black and Minority Ethnic (BME) Leadership for Higher Education Summit on Wednesday, organised by Advance HE, a new agency that will promote equal opportunities in universities.
Some of the examples of gaps in representation and achievement to be presented are:
- Out of more than 18,000 professors at UK universities, only about 110 are black
- Only 3.2% of academic managers, directors and senior officials are from ethnic minorities
- There is an average 16% pay gap between UK-domiciled white and ethnic minority academic managers, directors and senior officials
- There is a 15 percentage point gap between ethnic minority and white students getting first-class or upper-second degree grades
- There is a higher dropout rates for black undergraduates
Baroness Amos says that conversations about racism are difficult, because the debate often "degenerates" into arguments about whether individuals or viewpoints are racist.
But that misses the bigger picture of deep-rooted, "insidious", institutional prejudice.
"As a black person I know how hard it is to explain the pernicious and debilitating impact of day-to-day racism," she says.
"Many of us don't talk about it, but that doesn't mean it's not there."
There is much "rhetoric" about inclusivity and equality in universities, says Baroness Amos, which could cause "complacency" and a "slow pace of change".
"University leaders need to acknowledge that we are not doing enough," she says.
Nicola Dandridge, chief executive of the Office for Students, says the new watchdog recognises "gaps in outcomes between students in certain ethnic groups".
"Addressing these gaps, so that students from all backgrounds are able not just to get into higher education, but get on too, is a priority for the Office for Students," said Ms Dandridge, who will also address the conference.
There have been a series of recent claims of racism on campuses.
Earlier this month Exeter University expelled students after allegations of "vile" racist language online.
The National Union of Students has claimed that universities can be "more concerned about their reputation" than tackling racism.