A memorial garden could be built on the site of Grenfell Tower to honour the 72 people who died in the fire.
The 24-storey tower in west London was destroyed on 14 June 2017.
The Grenfell Tower Memorial Commission said bereaved families and local residents were struggling to agree on what should happen to the block.
It said families told them they would feel "deep pain" if it was to come down, while others said living in its shadow affected their mental health.
The report by the commission said it was engaging with survivors, the bereaved and local residents about ideas for the "community-led memorial", and that a garden was "by far" the most popular idea among those consulted.
Some 2,000 people were surveyed for the commission's report, including about 100 bereaved relatives and survivors.
The commission said 64% of those spoken to would like the memorial to include a garden, 33% would like it to include a monument or artwork, and 10% wanted a building, such as a museum. Some have suggested turning the tower itself into a vertical garden.
Other popular ideas were water features, a place to sit, and an area for children.
The commission said "further engagement" would take place with the community before a decision was made in 2023.
A design team for the final memorial is to be selected in 2024 and work could begin by the end of that year.
The commission said the government was responsible for deciding what happened to the tower itself, but a site of about 3,000 sq m (32,292 sq ft), including land where it stands, had been set aside for the memorial.
It is currently cordoned off to keep the building safe.
The report says that some bereaved families fear that it it comes down, "what happened will be forgotten".
"There's a worry that if Grenfell Tower no longer marks the west London skyline, the general public will think that the tragedy has closure and it's time to move on - and there will be less pressure to deliver justice," it said.
"For some, the tower should remain until prosecutions have been successful and there has been full accountability for what happened.
"At the same time, a large number of people - particularly those living and working in the local community - tell us that living with the tower places a huge strain on their mental health.
"Parents are concerned about how growing up literally in the shadow of the tower impacts on their children. It's not possible to know how long justice might take, and there are no guarantees that it will ever come. Waiting is costly in terms of their wellbeing."
The commission is made up of 10 community representatives, including five bereaved family members and three former Grenfell residents.
Andrea Newton, an elected representative on the commission, told BBC London the report was created to give people something to react to.
She explained the commission had spoken to groups linked with the memorials for the 9/11 and 7/7 terrorist attacks, as well as the Holocaust.
"They said how difficult it was and that is part of the process, having people find out they don't like things and then readjust and keep changing the idea until we arrive there - that is the process."
Survivor Miguel Alves said he hoped a decision would be made soon as he wanted to "look forward rather than be stuck in the middle". However, he added "justice should be done before the tower comes down".