"Ignorant" officials are obsessed with punishing victims of trafficking rather than targeting those behind the crime, a report claims.
A coalition of human rights groups says the system for handling victims in the UK is "not fit for purpose".
It wants an independent anti-trafficking watchdog to be created to oversee the work of officials.
The government says it will "look very carefully" at criticisms of the system "and act where necessary".
The Anti-Trafficking Monitoring Group is a coalition including Anti-Slavery International, Amnesty International UK and ECPAT UK (End Child Prostitution, Child Pornography and the Trafficking of Children).
In its report, "Wrong kind of victim?", it accuses the government of breaching its obligations under the European Convention Against Trafficking.
Amnesty International UK director Kate Allen said the emphasis must be "switched to victim protection, so that it's not all about hounding people for immigration offences".
"We pushed hard for the government to sign up to the anti-trafficking convention, but the government has botched its attempt to deal with this most despicable of crimes," she said.
"In particular, the identification system is clearly not fit for purpose, with under-trained staff displaying ignorance over what trafficking actually is."
The report criticises UK Border Agency staff for their treatment of victims.
In one case, it says a West African woman was told that although it must have been "extremely unpleasant" to be forced to have sex with strangers, this did "not amount to trafficking" because she had failed to escape when she had the opportunity.
In another instance, it says a woman forced into domestic slavery was told that as this had happened in 2008 she should now have "overcome any trauma".
Anti-Slavery International director Aidan McQuade said: "While the current system is undoubtedly staffed by many committed individuals, it is important to end the culture of viewing trafficking as a form of immigration crime.
"This focus has led to arbitrary decisions based on a dubious understanding of trafficking."
The report says poor treatment of victims discourages them from giving evidence against their traffickers, thus undermining efforts to secure prosecutions.
In the nine months to January 2010, it found that only 36 individuals were brought to court, despite the government estimating that some 5,000 trafficked people are currently in the UK.
Child victims of trafficking are also being failed by the system, the report says, and in particular, are not being provided with legal guardians to protect their interests.
ECPAT UK chief executive Christine Beddoe said: "Children are not mini-adults and attempting to fit them into a system designed for adults is inappropriate."
The report says that of 527 cases referred to the identification scheme between April and December 2009, 76% of UK citizens were positively identified as trafficked, compared with 29% of EU nationals and 12% for non-EU nationals.
It wants an independent watchdog to consider whether any discrimination may be taking place, and to oversee the system as a whole.
Immigration Minister Damian Green said: "Combating human trafficking is a key priority for the new coalition government.
"But our response must be about far more than law enforcement - identifying and protecting victims of this terrible crime is absolutely fundamental to tackling trafficking.
"I will look very carefully at the individual criticisms of the system set up in 2009, and act where necessary."