There is "overwhelming evidence" that the costs of HS2 are "out of control" and its benefits overstated, the deputy chair of its review panel has said.
Lord Berkeley said the high-speed rail line, linking London and northern England, is likely to cost over £108bn.
A vocal critic of HS2, the Labour peer said he believed MPs had been "misled" about the price - set at £55bn in 2015.
He has published a "dissenting report" on the project, but the government said it represented his personal view.
And a coalition of northern political leaders and businesses has also rejected Lord Berkeley's criticism of the project.
"We need HS2 and Northern Powerhouse Rail delivered together, in full," said the leader of Manchester City Council Sir Richard Leese, part of the Connecting Britain campaign.
Trains are due to start running on HS2 between London and Birmingham in 2029.
However, Lord Berkeley says there is little prospect of that before 2031, and warns high-speed trains will not reach Manchester and Leeds until 2040.
He told the BBC that spending money on improving rail services in the north of England was far more important.
"That's where the really bad quality railways are," Lord Berkeley said, adding that a complete upgrade "could probably be done at half the cost of HS2".
Lord Berkeley was the deputy chairman of the independent Oakervee Review in to HS2, set up by the government.
However, he has withdrawn his backing from the review, which is expected to be published in the coming months.
He says he disagrees with a draft version of its official report and as a result has published his own version.
He said he and other members of the panel were prevented from contributing to the final draft of the government-commissioned report because the review was "effectively terminated" on 31 October.
He also said the official report has been "unduly influenced" by HS2 promoters, and said both HS2 Ltd and the Department for Transport failed to co-operate with the review properly to substantiate their claims about the cost and benefits.
HS2 Ltd denies this claim, saying it provided "full co-operation" with the review.
His report concludes that ministers will either have to accept the higher cost of the line or only build part of the proposed high-speed rail network.
This option, which would save the government £50bn, would also involve upgrading existing Network Rail lines in the Midlands and the north of England, he said.
"The aim must be to give these areas the same standard of commuting service as the south east, whilst, at the same time, improving the existing lines from London northwards," Lord Berkeley said in the report.
He rejected suggestions that the increased costs were due to delays in the project, telling the BBC on Sunday: "The real cause is [HS2] has been over-designed. You do not need to go 400kph in a country as small as ours... The higher the speed makes a big difference to the cost."
He added: "People want a reliable, frequent service on which they can get a seat."
Anti-HS2 campaigners welcomed the report. "The case for HS2 has always been poor, and is simply getting worse," said Penny Gaines, chair of Stop HS2. "It is time for this white elephant of a project to be cancelled as quickly as possible."
And Greenpeace said that if the government was serious about climate change then it must listen to HS2's critics. "The protection of ancient woodlands must be a priority for rail development," said John Sauven, executive director of Greenpeace UK.
'Value for money'
However, HS2 Ltd, the company responsible for building the new line, says the economic case is strong and it is determined to deliver a railway that is good value for money.
A spokesman said the high-speed line was "critical" for the UK's future low-carbon transport network, would increase rail capacity and was "integral" to improving the rail network in the Midlands and North of England.
That view was backed by Sir Richard, who helps lead the Connecting Britain campaign. "After decades of underinvestment in strategic rail infrastructure, this is a once-in-a-generation chance to transform capacity and connectivity and level-up communities across the North, and beyond."
And he added: "We don't much appreciate being told by a peer, who divides his time between London and Cornwall, what the North wants."
A Department for Transport spokesman said: "The government commissioned the Oakervee review to provide advice on how and whether to proceed with HS2, with an independent panel representing a range of viewpoints... Lord Berkeley's report represents his personal view."
Asked whether the government was still committed to building HS2, Foreign Secretary Dominic Raab told Sky's Ridge On Sunday programme: "Yes, the prime minister's made it clear and we've got a review under way.
"We want to look at the best way to get value for money in relation to that substantial investment.
"We also want to make sure that we've got the best benefit from it in terms of the connectivity - not just in the South but in relation to the east, west in the northern region."
Members of the review panel told the BBC in November that a draft version of the report recommends that HS2 should be built with only minor alterations.
They include reducing the planned number of trains per hour from 18 to 14, in line with other high-speed networks around the world.
According to the Times, the draft report also says that "large ticket price rises" would be needed if HS2 does not go ahead, to prevent excessive demand for travel at peak times.