Councils who fail to speedily place children in their care with adoptive parents are being named and shamed in new performance league tables.
Currently, local authorities in England are required to organise adoptions for children within 12 months of deciding to do so.
But the process is complex, and children wait an average of two years seven months to be adopted.
David Cameron is promising tough action on those who fail.
This will include enforcing existing powers to strip councils of their responsibilities for adoptions.
The new performance tables rank local authorities on the speed at which their adoptions take place. They also rank them on 14 other measures of how well they are doing on caring for and improving the life chances of children in their care.
The Prime Minister said: "It is shocking that of the 3,600 children under the age of one in care, only 60 were adopted last year - this is clearly not good enough.
"So we will publish data on how every local authority is performing to ensure they are working quickly enough to provide the safe and secure family environment every child deserves."
Published for the first time on Monday, the adoption tables show that Hackney has the worst record.
In the east London borough only 43% of children are placed with adoptive parents within 12 months of a decision to do so.
Alan Wood, director of children's services in Hackney, said speedy placements should not be the only consideration upon which authorities are judged.
"We have got one of the best records of stability of placement; hardly any, if any, of our placements ever break down."
He added that Hackney was the fourth best in the country for the educational performance of looked-after children.
Other slow performers include the London Borough of Brent at 52% and Nottinghamshire at 55%.
At the top of the tables is York - the only council which managed to place 100% of children with adoptive parents within the 12-month time frame.
The average age at which children are adopted is three years and 10 months.
But many experts say adoption cases can be very complicated and need time to be worked through thoroughly. They can also be held up by delays in the courts.
England's adoption adviser Martin Narey, the former head of Barnardo's, is working with local councils to help them improve their adoption services. This includes overhauling the assessment process for those wishing to adopt.
Children's Minister Tim Loughton suggested if councils were not performing well enough services would be privatised.
"If they're not taking notice of us around a whole range of areas in terms of getting more children adopted, speeding the whole process up, making sure they're doing better by children in care and their outcomes, then we will want to put a very strong spotlight on them and say 'Are you really the right one to be running this service?'"
However, the Association of Directors of Children's Services said there were a number of alternatives to adoption which councils were increasingly using.
Its president Matt Dunkley said once these were taken into account, the numbers of children finding a suitable stable placement were rising.
"We agree that there are changes required to the adoption process to speed up the recruitment and matching of vulnerable children with potential adopters, as well as the decision that children should be put up for adoption, but not at the expense of depth and quality of decisions that risk adoption breakdown."
The association argues that these delays are as much a consequence of the court system that demands expert witnesses and endless assessments, as they are about problems in local authorities.
Hugh Thornbery, strategic director of children's services at charity Action for Children, said: "With the number of children in the care system at an all-time high, our need for adopters is greater than ever before."
Andrew, who is due to complete the process of adopting a daughter on Wednesday, told BBC Radio 4's Today programme that it took him and his wife two years to be matched with a child after they were approved as adoptive parents. He is Irish and his wife is of mixed race.
"Often we were told they were waiting for a more acceptable ethnic match. On one occasion there was a little girl who was part West African, and we were turned down because we had no West African connection," he said.
Chairman of the Local Government Association's Children and Young People Board Councillor David Simmonds acknowledged there was a variation in performance across councils and recognise that at times the system has been risk averse.
He added: "We want to work with government to change that and remove barriers that delay decisions, including tackling the significant delays in the family courts."
However, the BBC's Reeta Chakrabarti said: "Ministers have focused today on speed. But with one in five adoptions breaking down, support for families in the months after they adopted a child is critical too."