Parliament could burn down "any day", former minister Andrea Leadsom has warned as she urged MPs to "get on" with the renovation of the building.
Speaking to the BBC, she said the Houses of Parliament could see a fire similar to the one that damaged the Notre-Dame cathedral in Paris.
Parliament needs urgent repair work that could cost between £7bn and £13bn.
A recent report said costs could be kept down if MPs and peers left while the building work was carried out.
However, some politicians have expressed concern about moving out and plans to relocate to Richmond House in central London were vetoed.
In 2019 a major fire engulfed Notre-Dame, destroying the 850-year-old building's Gothic spire and roof.
Speaking to the Radio 4's Week in Westminster programme, Ms Leadsom said the same thing could happen in Parliament, adding that a fire in 2017 was only averted because of the presence of the 24/7 fire patrol.
"That is so 'there but for the grace of God'," she said.
"It could burn down today, tomorrow, any day and we've got to make that decision and get on with it."
The Houses of Parliament - which was built between 1837 and 1860 - requires significant restoration, including asbestos removal, fire safety improvements, renewal of wiring and conservation work.
In addition to a fire risk, there have been warnings the building could be damaged by a flood of sewage. A sewage ejector system installed in 1888 is still in use now.
In 2019, MPs voted to set up a sponsor body with responsibility for the restoration of the building.
A review produced by the body put the basic costs of essential repairs at £7bn-£13bn and estimated they would take between 19 and 28 years to complete - or 12 to 20 years if the building was fully vacated.
It said the costs would increase by 40% if politicians stayed put.
Expressing concern about the project's timescales and costs, the commissions - or administrative bodies - for the House of Commons and Lords voted to scrap the sponsor body in favour of a "new approach".
The commissions also agreed to seek approval from MPs for a "revised mandate" for the project.
Politics Professor Matt Flinders of Sheffield University has previously suggested the change was the result of a "massive blame game" because the politicians were shrinking from an unpopular decision.
Last week, Levelling Up Secretary Michael Gove rejected suggestions the House of Lords could move to the Queen Elizabeth II Centre in Westminster while building work was under way.
Instead he said peers should consider moving to alternative buildings in the North of England, the Midlands, the South West, Scotland or Wales.
But crossbench peer Baroness Hayman claimed Mr Gove's suggestion was "punishment" for the House of Lords voting down parts of government bills.
You can hear the full interview with Andrea Leadsom on the Week in Westminster on Radio 4 at 11:00 BST on Saturday 21 May.
- 'I LEFT THREE KIDS AND A WIFE I ADORE': Ukrainian tennis player who returned to defend his country
- 'I EARN MORE THAN MY FRIENDS WHO WENT TO UNI': Meet the young bricklayers building their lives