The amount of money spent on services for teenagers in England has fallen by 36% in the past two years, according to figures released to the BBC.
Former children's minister, Tory MP Tim Loughton, said the £438m reduction in spending was "disproportionate".
Youth workers warned that the long-term cost of the cuts would be "enormous".
But the Local Government Association said funding cuts meant there were "no easy choices" and spending on things like child protection came first.
The figures, released to BBC Radio 4's World at One after a Freedom of Information request to the Department for Education, outline the amount spent by local authorities on providing services like youth clubs and other out-of-school activities.
The spending also covers education for excluded pupils, teenage pregnancy services and drug and alcohol support programmes.
They show that, in real terms, the amount spent by councils fell from £1.2bn in 2010-11, to £791m in 2012-13.
The biggest cut in percentage terms was in the London borough of Kensington and Chelsea which reduced its budget by 78%, or £5.1m, while Tower Hamlets cut spending by £9.4m - a 65% reduction.
Outside the capital, Tameside, Stoke-on-Trent and Warrington all cut spending by more than 70%.
The amount being spent increased in seven out of 152 areas, including Oldham and Hertfordshire.
Tory MP Tim Loughton, who was children's minister until September 2012, said "councils clearly are cutting youth services disproportionately".
He said a requirement that councils must provide "sufficient leisure-time activities" for teenagers, but only "so far as reasonably practicable", meant youth services were a "soft touch".
"Because they don't have to statutorily provide youth services they have too often been at the top of the queue when cuts come along," he said.
Mr Loughton also described a decision to move responsibility for young people from the Department for Education to the Cabinet Office as a "retrograde step".
He said youth policy "should be back in the Department for Education where you've got that clear interface with what young people do in schools".
Fiona Black, chief executive of the National Youth Agency - the national body for youth work, said the cuts will lead to problems in the long term.
She said: "We're going to see more young people in the criminal justice system, more young people who perhaps aren't engaging in education. The cost of that to taxpayers is enormous compared to the very small investment in youth services."
But David Simmonds, who chairs the Local Government Association's children and young people's board, said councils faced "no easy choices".
Some councils, he said, "have been badly affected by the level of reductions in government funding and that's meant we've seen some areas where the level of funding going into youth services has gone down really quite substantially".
He added that councils had to prioritise some services over others.
Mr Simmonds said: "The government has made some decisions about how to respond to the overall austerity situation.
"Councils are faced with rapidly rising demand, in particular for child protection services. So in order to fund that we need to look at the things that have a less direct and less immediate impact on the lives of children and young people."
Update 2 April 2014: Kensington and Chelsea points out that the reduction in its funding is because youth services have been spun off in a free-standing staff mutual which is funded separately.