The opportunity for the London Olympics to inspire a generation of children to participate in sport has been squandered, Tessa Jowell has said.
The former Olympics minister, who is standing to be Labour's candidate for London mayor, said cuts to school sport money had been "wicked and negligent".
But she said the regeneration of east London, a key pledge, was a success.
The government said 1.4 million more people were playing sport each week than when London's Olympic bid was won.
But a spokesman for the Department for Culture, Media and Sport said more needed to be done to attract people from all backgrounds.
Dame Tessa's comments come exactly 10 years after London was awarded the Games in 2005. She was Olympics minister when the decision was announced.
She told BBC Radio 4's Today programme: "My particular concern is that the second legacy promise was to transform a generation of young people through sport.
"Actually children in school now are playing less sport than they were in 2009.
"That's why it is a wasted opportunity."
A Youth Sport Trust survey of 1,392 primary schools in 2013/14 found that the average number of minutes children spent taking part in PE in a typical week was 102 for Key Stage 1 and 114 for Key Stage 2.
This was down from 126 minutes for Key Stage 1 and 127 minutes for Key Stage 2 in the charity's 2009/10 survey.
Dame Tessa criticised the government's decision to cut funding for school sport. In 2010, £162m of ring-fenced funding for the national School Sport Partnerships (SSPs) was abolished.
"We had a model of organisation that was the envy of so many other sporting countries around the world, that provided the organisational framework to make sure that the matches were organised, notes went home so parents knew when their kids were coming home," she said.
"That structure of organisation has been dismantled. And the evidence is that if you don't tell schools that the money for sport is dedicated to sport, it's spent on other things."
The former MP for Dulwich and West Norwood told the Guardian that "the most wicked and negligent part of it was winding up school sport partnerships".
She added: "I hope this is a huge stain on their consciences."
Lord Sebastian Coe, who was chairman of the London Organising Committee of the Olympic and Paralympic Games, wrote in the Daily Telegraph: "Pre-2005 there is little doubt sport participation was declining and had been for a long time, particularly amongst young people.
"Since bidding for the Games, that trend has reversed."
Sir Keith Mills, who was on the organising committee for the London Olympics, told the BBC the government had to do more to encourage school sport.
He said: "The new sports minister is now talking about developing a national strategy for sport and I think that's something I'd encourage the government to do - bringing together the different departments, education, health and communities and treasury and others - to really provide a bottom up, joined up approach to sport."
Dame Tessa, who remained shadow Olympics minister after Labour lost power in 2010, was more positive about the impact the Games have had on east London.
"The regeneration has been extraordinary - a whole new park in east London, venues which are used both for elite sport and for the community, so that is absolutely the case," she told the BBC.
She said more homes had been built and more jobs created.
A spokesman for the Department for Culture, Media and Sport said: "We are investing over £1bn of public funding into grassroots sport over five years and there are 1.4 million more people playing sport each week than when the Olympic bid was won.
"More needs to be done to attract new people from all types of backgrounds to participate in a variety of sports - which is why the government is working on a new sports strategy to challenge sporting bodies to deliver on the public money they receive, and to strengthen community sport across the country."
Sport England's latest figures show the number of people over 16 participating in sport at least once a week stood at 15.49 million in April 2015, down from 15.89 million in October 2012, just after the Olympic and Paralympic Games finished.
Its statistics show that 3.9 million young people aged 16 to 25 participated in sport once a week in October 2009, declining to 3.8 million in April 2015.