The Olympic Stadium is not being used as well as it should be, according to a group of Lords who also questioned the legacy of London 2012.
The House of Lords Committee on Games Legacy said the £429m stadium was a "national asset that should be used to the full".
But it remained "unconvinced" that present arrangements would deliver an "effective and robust" legacy.
The government said it remained committed to building on the Games.
While acknowledging that the Olympics and Paralympics were an "outstanding success" the report also calls on the government to appoint a minister to take overall responsibility for delivering legacy benefits.
The major examination of London's legacy efforts by a cross-party committee of peers claims political impetus has been lost and there is a "lack of ownership" from the government over building on the opportunities the Games provided.
The report's authors say there are no white elephants among 2012 facilities but there is "little evidence" that a step-change in sport participation levels has materialised - a key pledge from London's bid for the Games.
There is also criticism over the geographical spread of the economic benefits - with foreign investment as a result of London 2012 resulting in 15,000 jobs for the south-east, but just seven in the north-east.
And it said there may have been too many "closed minds" about potential options for how the Olympic Stadium could be used.
West Ham United was chosen to be the anchor tenant of the 80,000 capacity venue, which is set to be turned into a 54,000-seater stadium in time for the 2016-17 season.
The east London club has been awarded a 99-year lease on the stadium in Stratford.
This summer, the venue staged a sell-out athletics meeting on the weekend of the first anniversary of the start the 2012 Olympics.
Pop concerts, 2015 Rugby World Cup matches, the 2017 World Athletics Championships, and future London Diamond League athletics meetings are all set to take place there after the reconstruction work.
Essex County Cricket Club has suggested it could play Twenty20 games at the stadium and earlier this year a bid to host the 2022 Commonwealth Games was mooted.
But last November the chairman of UK Athletics, Ed Warner, said he had expected the stadium to reopen in 2014 and branded the delay to 2015 or 2016 "a farce".
Leyton Orient wanted to share the use of the stadium and the League One club had applied at the High Court to have a judicial review of the decision to award it to the Hammers but this was refused.
'Squabble not helpful'
Lord Harris, chairman of the Lords committee, said the clubs were acting "like children squabbling over who goes down the slide first as far as the stadium is concerned".
"We urge those concerned to think further on how the two most local football clubs might work together, including whether any difficulties can be ameliorated through wider community use of the stadium, which may include its occasional use by Leyton Orient FC if appropriate financial arrangement can be agreed," he said.
In its report, titled Keeping The Flame Alive, the committee said: "Our feeling is that the stadium is a national asset.
"There has been very substantial investment in the facilities. Given that this is a national asset and the focus should be on making the best use of it for the community, this squabble is not helpful.
"At the moment our concern is that there are too many closed minds about this."
Lord Harris added: "At the moment I think all we have heard is people's negotiating position rather than a serious attempt to reach a compromise."
Leyton Orient welcomed the report. Chairman Barry Hearn said: "I agree with the House of Lords recommendation - ourselves, West Ham and the LLDC should sit down and work this out together once and for all."
The club said it would seek a meeting with the LLDC to discuss "what part we can play in the future use of the Olympic Stadium".
A spokesman for West Ham said the club was focused on "creating a stunning new home for the club and its supporters in 2016, alongside a long-term legacy for the community of east London".
"What goes on with other interested parties is very much a matter for (Leyton Orient) and the LLDC, and not West Ham United," he said.
'Blueprint' for future Games
The committee also said time frames and targets were confused and the political drive which helped to make the Games a success had now fallen by the wayside.
As well as suggesting the appointment of a legacy benefits minister, the peers also called for the Department for Transport to try to secure international train services at Stratford International station, in order to generate a return on the substantial investment made there.
A Department for Culture, Media and Sport spokesman said: "The government remains committed to building on the legacy created from hosting the 2012 Olympic and Paralympic Games to benefit the entire country for generations to come.
"Our legacy has been hailed by the International Olympic Committee as a 'blueprint' for future Games and over the next 10 years we aim to use the inspiration of 2012 to deliver lasting change in sport, communities, the economy, east London and awareness and perceptions around disability."