Councils in England are holding on to £1.5bn of unspent "community money" given to them by developers during planning talks.
The Section 106 money is intended to be used for social and community projects.
But at a time of council cutbacks, £421m of it has not been allocated to any future schemes.
And over the past five years £9.8m has been returned to developers, often because it has not been spent within a set time period.
Freedom of Information requests were submitted by the BBC to all 353 local authorities in England. Of the 316 that responded:
- Hertfordshire County Council holds the most "unspent" money with £56m
- Swindon Borough Council has the most "unallocated" money with £18m
- Essex County Council returned the most cash to developers, with £1.2m in the past five years
Section 106 agreements are negotiated between a developer and council to help make new home schemes more attractive to communities.
It can be used to help fund affordable housing as well as roads, parks and youth services.
But there are concerns that at a time of council cutbacks the money is not being handled properly and being left too long before being spent.
Mark Dempsey, who is the shadow deputy leader of the Conservative-controlled Swindon Borough Council, where £18m has not been allocated to any specific projects, said: "It's a story of not managing section 106 money. What we need is a full review of how this money is managed, so we can give real value for money."
The Labour councillor said a development in the north of the town led to Section 106 money being returned to developers.
"It was meant to be spent on a new link road to the town centre, a new sports hall and football pitches, but none of this has taken place," he said.
However, a council spokesman said Swindon was one of the fastest-growing towns in the UK with well over a thousand homes being built each year on average.
"This inevitably means that the council has a significant amount of Section 106 money which is waiting to be spent at the right time on specific schemes," he said.
"Although more than half of the council's Section 106 money has already been allocated to specific projects, there are many reasons why it isn't always possible to do this quickly."
He said sometimes the money is paid in instalments meaning it cannot be allocated to a project until it has all been received.
But at a time of deep cuts, many groups have called for the Section 106 money to be used to alleviate the pain of austerity.
In Essex there are concerns that youth services will be drastically cut from £5m to £2m, despite the Conservative council having £15.5m of unallocated money.
It has also returned £1.2m to developers over the past five years.
Mike Hibbs, chairman of the Saffron Walden Youth Outreach Project, said: "The Section 106 money could be put to good use - there's no doubt about that."
He said the shortfall of funding for youth services could have been met with Section 106 money.
"It's very sad - generations of young people have benefitted from community services," he added.
John Jowers, who is the cabinet member for libraries, communities and planning at the county council, said the authority had received more than £36m in contributions. He added that it was within this context that the return of funds to developers must be seen.
"We try as far as possible not to return contributions but we acknowledge that there are circumstances in which returns are inevitable," he said.
He said these include when money was not spent within a particular time frame, when funding was more than adequate to meet the final costs or when the infrastructure was no longer needed.
A spokeswoman for Conservative-run Hertfordshire County Council, which has returned £691,000 and has £58m of unspent Section 106 money, said there are "strict legal limitations" on how the money can be spent.
She said: "Money provided through Section 106 agreements can only be used for the purposes set out in the agreement and not for other purposes.
"Agreements often include geographical limitations, and limits round how long the money may be kept for."
John Stewart, director of economic affairs at the Home Builders Federation - which represents developers, said local authorities "clearly have an obligation to spend these funds to benefit their communities."
He added: "However it is also imperative that they transparently account for these contributions to their residents who must see and experience the true benefits that come with the building of new homes in their area.
"Otherwise residents will simply see new housing placing additional burdens on local facilities and services, unaware of the compensating benefits from S106 agreements, whether these are purely financial contributions, or benefits in kind."
Mike Jones, who is the chairman of the Local Government Association's environment and housing board, said the process behind assigning the Section 106 money was complex.
"It is utterly wrong to say councils are 'sitting on it'," he said.
"This is all about making the right long-term investment decisions to deliver the services that communities desperately need."
Are you waiting for a playground to be built at a new development near you? Email us at firstname.lastname@example.org with your stories.
Follow BBC News England on Twitter@BBCEngland.