The disturbance of a "part of village history" has sparked anger among residents.
The Popping Stone - one large and two small boulders on a path through the Irthing Gorge at Gilsland on the border between Cumbria and Northumberland - has long been a tourist attraction.
It was so named because it was where Sir Walter Scott reputedly "popped the question" to his beloved in 1797.
However, the path was recently widened and the stones were "messed about".
Many country walks feature the Popping Stone, and it has been a visitor attraction since a rail line opened between Newcastle and Carlisle more than 100 years ago.
Although it does not have listed or protected status, Historic England described it as being of "historical interest".
The land was in the grounds of a hotel, and after that business failed it was sold off in parcels.
At the weekend walkers reported seeing tracks of what appeared to be a digger or bulldozer, and the stones had been damaged. It is unclear who was responsible.
Malcolm Redman, who runs a nearby bed and breakfast, said: "The stones seem to have been disturbed and dug over and messed about.
"While they may not have legal or historical designation, they are part of village history.
"They've attracted generations of locals and visitors."