Social workers should not delay placing a child with a suitable family of a different ethnicity, ministers say.
New adoption guidelines for England aim to break down the barriers faced by people looking to adopt children in England.
Education Secretary Michael Gove said too many were denied loving homes.
He said some of the limitations put on adopters in the past - based on sexual orientation, ethnicity and faith - was "social engineering of the worst kind".
In the past, many children from ethnic minorities do not get adopted because social workers have been keen to place them with families of the same background.
Long wait for adoption
The government said progress in adoption had stalled in recent years, with the number of children placed for adoption falling by 15% between March 2009 and 2010, and more children waiting longer to be adopted.
It said black children took on average over 50% longer to be placed for adoption than children from other ethnic groups.
And children aged over five were four times less likely to be adopted compared to children under five.
The law will not change but the new guidance will state that as long as prospective adopters show that they are able to care for the child then race should not be a factor.
They will also say that preventing families from adopting children of a different ethnic group is "unacceptable".
Each local authority will be closely monitored and those that persistently ignore the advice could have their adoption services contracted out to voluntary agencies.
Current advice states that social workers must give "due consideration to the child's religious persuasion, racial origin and cultural and linguistic background", but does not specify whether race should be regarded as outweighing other factors.
Interests of children
Mr Gove said thousands of children were currently in the care system waiting to be adopted.
"Every day they wait is a day they're denied the loving home all children deserve. But politically correct attitudes and ridiculous bureaucracy keep many of those children waiting far too long.
"What I do find difficult to accept is that we've created over time a web of rules that mean that we are not always putting the interests of children first.
"We all know that the length of time which children spend in institutional care once they be been taken into care is far too long, when those children could be adopted by loving parents."
He said the average amount of time children spend in care before being adopted is 21 months.
Children's minister Tim Loughton announced in November last year that the government would be updating its guidance on adoption.
Currently, single people, married couples and cohabiting couples can all adopt.
Same sex couples can adopt in England, Wales and Scotland, but not in Northern Ireland.
Adopters in England and Wales must be aged over 21 (18 if one of a couple is the birth parent). There is no upper age limit, but the placing authority must be confident anyone adopting a child will have the energy required and be in good enough health to offer a stable home.
Likewise, if they can prove capable, people will not be disqualified for being disabled, overweight or having a medical condition.
People must also show that the can financially support a child.
A criminal record will not automatically prevent someone from adopting unless they, or someone in their household, has been convicted or cautioned for offences against a child.