Setting targets to hire judges from ethnic minority backgrounds would be the "wrong way" to solve the issue of diversity, says the justice secretary.
David Lidington has outlined steps to tackle "race bias" in the legal system in England and Wales, following a damning report from MP David Lammy.
But the minister says the government needs to "look at the critical path" into the law - rather than targets.
Mr Lammy said he was "disappointed" with the government's response.
The Lammy Review, published in September, said that people from black, Asian and minority ethnic backgrounds make up 25% of the prison population and 41% of the youth justice system - but only 14% of the general population.
Ethnic minority groups make up only 11% of magistrates and 7% of judges.
As a result, Mr Lammy had called for a national target to achieve representation in the courts by 2025.
However, Mr Lidington said a target would be "the wrong way to attack this particular objective" and he was looking at alternatives.
He told BBC Radio 4's Today programme: "When you look at the judges, you have got a group of people who have been practising in law perhaps for 20 years... because we need people who are experienced, who are expert, to sit on the bench.
"In getting a more diverse judiciary... you need to look at the critical path of how do people get into the legal profession in the first place."
Mr Lammy disagreed and instead called for a "bold approach".
The Tottenham MP told the BBC: "It is not about the pipeline. BAME (Black, Asian, and minority ethnic) lawyers are applying to join the judiciary.
"If you set a target or a goal, then it concentrates the mind to achieve that. But the government has not affected that.
"The UK is behind the curve on diversity and it needs to catch up."
Lawyer and diversity campaigner Funke Abimbola was also disappointed with the decision, telling Today: "It is an important element of [the recommendations] because bias in the judiciary has a direct impact on decision-making.
"This whole report was about there being proven bias against those minority ethnic backgrounds. [Targets are] a key part in driving diversity of thought, which impacts decisions in court."
By BBC home affairs correspondent Danny Shaw
The government's response to David Lammy's proposals suggests it's following them in spirit - but not to the letter.
Many of the recommendations relate to changes to data collection, which should be relatively straightforward to implement.
The harder task set by Mr Lammy involves increasing diversity, so those who work in the criminal justice system better reflect and understand the people who use it.
Although ministers have agreed challenging targets to raise ethnic minority representation among prison officers, they've balked at doing the same for judges and magistrates.
The experience of the police service, which failed to meet 10-year BME targets set in 1999 after the Macpherson report, may have influenced their decision.
Targets can help focus efforts in achieving a goal - but if they're too stretching, they may prove counter-productive.
'Change or explain'
The Lammy Review concluded that people from minority backgrounds still faced bias, "including overt discrimination", in parts of the justice system.
Mr Lidington pledged to implement a "key principle" of "change or explain" when racial discrimination is found in the system.
"Where we cannot explain differences in outcomes for different groups, we will reform," he said, pledging to work on each of Mr Lammy's 35 recommendations - even if not following them to the letter.
And he added this was the "very first step" in a change of attitude towards race disparity "that will touch on every part of the criminal justice system for years to come".
Within the government's response, it said it would:
- Collect wider datasets related to ethnicity for more accurate analysis
- Publish more datasets so the public can see the reality
- Review use of "gang" prosecutions for those who are victims and exploited by adults
- Review modern slavery legislation to see how it could help with gang crime
- Redact, where possible, identifying information to make "race-blind decisions"
- Investigate better ways of explaining legal rights and options to build trust
- Assess maturity of young offenders over 18
- Recruit a more diverse workforce in prisons and help people move to more senior roles
- Consider sealing criminal records
- Trial deferred prosecutions - where offenders can avoid trial when admitting the offence - in London
Some of the changes would take longer to achieve than others, the Ministry of Justice said.
But it had already made progress on several recommendations, including publishing data on race bias in the system.
It said a new race and ethnicity board would drive through the reforms - but alternative approaches would be found where proposals could not be implemented in full.