The collapse of Lewes Castle's boundary wall has been described as "catastrophic" by a Historic England inspector.
Rebecca Lambert said she was "deeply saddened" by the news and the group would be "offering advice and support" to those responsible for the wall.
About 600 tonnes of rubble fell from the 11th Century structure into a neighbouring garden on Monday.
East Sussex County Council said the site was under "multiple ownership".
Firefighters used sniffer dogs and a drone to search through the rubble amid fears people could be trapped.
Neighbour Harriett Wyndham said emergency services became concerned when a discarded yellow vest was found in the garden.
No casualties were found.
Following the collapse, Ms Lambert tweeted "days like this are tough".
Days like this are tough, such a catastrophic collapse - deeply saddened at this news, but we will be offering advice and support https://t.co/zhN5omn3bh— REBECCA LAMBERT (@IndianaCroft1) November 11, 2019
Dr Matthew Pope, senior research fellow at the Institute of Archaeology at University College London, said it was "a serious incident to learn from".
Shocking image of the collapse of a privately-owned section of Lewes Castle's curtain wall. Thankfully it is not thought anyone was hurt, but this is a serious incident to learn from. Sobering to see how catastrophically these fortifications can collapsehttps://t.co/t2wv9huPGb— Dr Matt Pope (@MatthewPope) November 11, 2019
While heritage tourism researcher Coralie Acheson described it as a "nightmare".
Gosh - it sounds like there’s probably no one hurt (hopefully). This is my general nightmare - that I won’t be able to talk an inspector of ancient monuments/conservation officer round fast enough to save someone’s life and it will be my fault. https://t.co/ERpAjRTfCE— Coralie Acheson (@Coralfrog) November 11, 2019
Ms Wyndham said she thought a train might have derailed when she heard a "rumbling sound" from her kitchen.
She said it was "quite a shock" when she looked out of her window to see "a vast amount of dust, then a vast amount of rubble".
"It was fortunate the owners [of the house] were not here, in their garden," she added.
A spokesperson for the county council said responsibility for sections of the wall had been transferred to it in 1975 and it would be working closely with the affected households and emergency services to assess why it collapsed.
"The county council undertook survey, repairs and strengthening works to this section of the wall in 2017 under Scheduled Monument planning and building approvals, but we are not aware of any more recent activity that may have led to this collapse," the spokesperson added.
Lewes District Council Green Party Councillor Roy Clay called for a full report by structural engineers to explain the collapse.
He added: "Following on from that, we might need to look at the other walls in Lewes around the castle and elsewhere to make sure it doesn't happen again."
You may also be interested in:
A spokeswoman for castle owners Sussex Archaeological Society said it was one of the last remaining sections of the historic defence wall that fell.
She added: "There's not much of it left standing. It's one of the last sections that was still at full height."
Earlier, Lewes Council tweeted that the South Tower remained closed for safety checks.
Emma O'Connor, from Sussex Archaeological Society, confirmed the closed tower was of a similar construction to the wall and would therefore remain shut until a full inspection had taken place.
Historic England said it was in contact with Lewes District Council and has offered assistance "both in the immediate aftermath of this event and for the future, as proposals to rebuild will need to be considered".