Removing controversial statues risks the past being "torn away", the Historic England chair has said.
Speaking to MPs, Sir Laurie Magnus said contested heritage should be left standing but "recontexualised".
Appearing at the same committee Bristol's mayor Marvin Rees said it was "a difficult tightrope to walk".
The MPs' inquiry comes four months after protesters threw a statue of slave trader Edward Colston into Bristol harbour.
Mr Colston was a member of the Royal African Company, which transported about 80,000 men, women and children from Africa to the Americas in the 17th century.
A number of memorials to other merchants who profited from the slave trade were removed this summer during a wave of anti-racism protests in the UK, sparked by the murder of George Floyd in the US.
The subject was touched on by Prime Minister Boris Johnson in his speech to the Conservative party conference, in which he accused Labour of wanting "to pull statues down, to re-write the history of our country, to edit our national CV to make it look more politically correct".
Labour leader Sir Keir Starmer said in June that the way the Colston statue had been pulled down was "completely wrong" but it should have been removed "a long, long time ago".
'A lot of upset'
Giving evidence to the committee, Sir Laurie said that trying to "erase whole swathes" of history would lead to a "lot of upset and trouble" among "most people" across the country.
"Our collective past is going to be just torn away, slowly, piece-by-piece," he said.
He rejected the idea that contested statues should be put in a museum calling instead to "leave them where they are and recontextualise them".
"Our collective past is there, it represents a memorialisation going back hundreds of years, built at a time which reflected the views and values of those who lived at the time.
"If we start tampering with the historic fabric connected with our collective past because things are contentious, then you start changing the basis in which you can understand it.
"And I think understanding is really important."
He said he was looking to "find solutions" to ensure contested statues remain on display.
Historic England is a government body which advises organisations on preserving England's heritage and gives grants to groups for the conservation of buildings and landscapes.
Addressing the same group of MPs, Mr Rees said that Mr Colston was not simply controversial but "evil" and while he couldn't condone criminal damage, he acknowledged the statue was "an affront to me".
He also said the presence of contested statues "is an ongoing message to some people that they don't matter" but added that symbolic acts should also be "about real policy change".
He emphasised that the discussion on statues had to "include everyone" and told the committee how he had met with those who had gathered to "protect" Bristol's war memorial the weekend after the Colston statue was removed.