The government is promising to focus more on people's social class and individual "character", as it overhauls its equality policy.
Equalities minister Liz Truss said the discrimination debate should not focus solely on race, religion, sexual orientation and disability.
Discussion had too often been dominated by "fashion" and not "facts", she said.
Labour accused Ms Truss of "gratuitous provocation" and ignoring the "devastating impact" of discrimination.
The government is launching an Equality Data Programme to gather information on people's backgrounds, social mobility and inequality between regions.
Speaking at the Centre for Policy Studies think tank, Ms Truss said: "To make our society more equal, we need the equality debate to be led by facts, not by fashion.
"Time and time again, we see politicians making their own evidence-free judgements."
She also said discussion had "been dominated by a small number of unrepresentative voices, and by those who believe people are defined by their protected characteristic and not by their individual character".
The Equalities and Human Rights Commission lists protected characteristics - over which it is illegal to discriminate - as age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation.
'Real concerns people face'
Ms Truss argued that data based solely on these characteristics is not fit for purpose when it comes to setting equality policy.
The equalities minister said: "Underlying this [approach] is the soft bigotry of low expectations, where people from certain backgrounds are never expected or considered able enough to reach high standards.
"This diminishes individual humanity and dignity, because when you choose on the basis of protected characteristics, you end up excluding people."
Ms Truss added that it was "appalling" that pregnant women suffer discrimination at work", that women may "be encouraged to dress in a certain way to get ahead", and that "some employers overlook the capabilities of people with disabilities".
Debates on equality must be "rooted" in "real concerns people face", she said, adding: "It is our duty to deliver, because if right-thinking people do not lead the fight for fairness, then it will be led by those whose ideas do not work."
For Labour, shadow equalities minister Marsha de Cordova said: "This is gratuitous provocation from a government that consistently refuses to face up to its responsibilities and the widening inequality it has caused.
"When Liz Truss dismisses 'fashionable' causes, she actually dismisses the devastating impact of discrimination and unfairness in people's day-to-day lives."
Halima Begum, chief executive of the Runnymede Trust, a racial equality think tank, said: "Liz Truss's attempts to 'overhaul' the equalities work in the UK is nothing short of a whitewashing of British history and its relationship with race."
She also said: "It is time that equalities ministers in this government are held accountable for their words."
The government announced on Wednesday that its report into racial inequality will be delayed until next year, citing problems caused by the coronavirus pandemic.
It is looking at health, education and criminal justice, but also "wider inequalities" such as issues faced by working-class white boys.