Council spending on social workers from agencies has doubled in the past four years, figures show.
UK spending increased from £180m in 2012-13 to £356m in 2016-17.
Staff say relying on agencies to fill shortages in the permanent workforces can drain council funds and high turnover among these staff can lead to a poorer quality of service.
The Department for Education said it had invested more than £750m to support the recruitment of social workers.
Lesley Hagger, director of children's services at Northamptonshire County Council, said: "Whilst we are using our resources to pay high costs for agency staff, we can't provide other services we might want to, such as early help services, to prevent children coming into care in the first place."
And while she valued agency staff and they were very good, they cost more than she could afford to pay.
Councils struggled to recruit social workers, because they were being "lambasted", and this drove up agency costs, Ms Hagger said.
"They are the people that do one of the most important and difficult jobs in the country, and they don't get very much recognition for it," she said.
Why is there such a high use of agency staff?
By Alison Holt, BBC social affairs correspondent
The demands made on social workers are huge and increasing, whether they are protecting children or helping adults who are vulnerable because of their age or disabilities.
They are professionals who in case conferences will sit alongside doctors and police officers as life-changing decisions are made.
But many social workers will say rather than the respect those other professionals get, they face suspicion and blame if things go wrong.
They have long complained about poor pay, high caseloads and the unrelenting pressure of the job.
That's why working for an agency, where you get paid more and can leave when you wish, can seem like an attractive option.
But for those who rely on their help, such as children, frequent changes in social worker are damaging.
It makes it harder to trust the person there to help you, if you think they will leave soon.
And it is easier for warning signs to be missed when cases are handed from person to person.
The ongoing challenge for councils and government is not just how to recruit enough people to be social workers but how to make sure they will stay for the long term.
And with so much money being spent on plugging the gaps, the cost of failing to do this is massive.
The BBC sent Freedom of Information requests to all councils in the UK with social care responsibilities asking for their spending on agency social workers.
This covered staff working with both adults and children.
Of the 211 councils, 153 responded in full, showing spending had risen £176m over the past four years.
According to Department for Education figures, the number of agency social workers employed by local authorities in England's children's services rose from 3,250 to 5,330 between 2013 and 2016.
The BBC was able to collect data for only the first 11 months of 2016-17.
Our full-year figure assumes that spending in the final month of the year was not very different to the rest of the year.
Some authorities provided information for their whole children's and adult services departments, with their figures including social work administrators as well as qualified social workers.
Councils with largest agency social worker spending increases for adults and children's services from 2012-13 to 2016-17
- Northamptonshire: £17,392,364
- Birmingham: £12,248,660
- Wandsworth: £7,263,426
- Rotherham: £6,170,565
- Kirklees: £6,118,715
In Northamptonshire, spending on agency staff in children's services in particular rose from £1.7m to £17.9m over the four-year period
Sue Stevenson, team manager of the social work academy at Northamptonshire, said the resulting staff turnover made it very difficult to follow through with care plans for vulnerable children.
"Risk will get missed, potentially, and support not be put in place in a timely way, because maybe that referral has been made but then it might need chasing up," she said.
In an attempt to retain more permanent staff, the academy gives newly trained social workers additional training and guidance, as well as less complex caseworks in their first year.
Claire Kober, who chairs the Local Government Association's Resources Board, said: "The government needs to support council schemes to recruit social workers, by improving the image and career prospects of social work.
"It needs to provide more funding and give councils freedom to use this money to the best effect according to local priorities."
Unison head of local government Heather Wakefield said: "With morale so low, many social workers are choosing to move into agency work where they have more control over their hours and can get better pay.
"The government has made social work a profession in crisis."
The Department of Education said training programmes were bringing in high-calibre graduates and preparing them for the realities of both children's and adult social work.
It said more than 1,300 adult social workers had qualified in 2016-17.
"Councils increased spending on children and young people's services to over £9bn in 2015-16, and we are empowering councils to develop new and better ways of delivering these services," a spokesman added.