Prejudice has hampered attempts to build a statue to the Elephant Man, his biographer has claimed.
Jo Vigor-Mungovin, who traced Joseph Merrick's grave, has been trying to raise the estimated £100,000 cost for a monument in his native Leicester.
But progress has been slow with critics saying the idea was a "freak show" and the city was already "ugly enough".
Mrs Vigor-Mungovin said: "There is a fear of what the statue would be like - but he was an inspirational figure".
But the plans for the statue have been backed by artist and disability campaigner Alison Lapper who said anyone offended should "get over it".
Merrick was born in Leicester in 1862 but his physical disabilities forced him into a workhouse.
In 1884 he joined a travelling exhibition and eventually found his way to the Royal London Hospital, where he died in 1890.
While Merrick's skeleton was kept at the hospital, his soft tissue was buried in a common plot but its location was forgotten.
Shortly after tracking down the site of his grave, Mrs Vigor-Mungovin began looking into erecting a statue to Merrick.
"I wasn't expecting it to be controversial," she said. "But I've come across the same reaction over and over.
"When I approach funding sources or venues, people seem interested at first but when they hear it will be a statue of the Elephant Man, they seem a bit shocked.
"They either say 'you can't do that' or stop answering emails or the phone.
"I'm a descendant of Tom Norman, the showman who worked with Joseph, and I've even been accused of being an accomplice to a 'Vampire showman's crime'."
- Born in August 1862 in Leicester
- His condition did not develop until he was five years old
- Following years in a Leicester workhouse, he contacted a showman who set him up as a travelling exhibit in 1884
- After being robbed and abandoned he went to London in June 1886 and contacted Dr Frederick Treves, who found him a room at the London Hospital
- His head measured 36 inches (91cm), his right wrist 12 inches (30cm) and one of his fingers 5 inches (13cm) in circumference
- He died on 11 April 1890, aged 27, asphyxiated by the weight of his own head, apparently after trying to lie down
- The cause of his condition is still uncertain but many researchers favour Proteus syndrome, a rare genetic disorder
- His story has inspired a number of books, an award-winning play and a film starring John Hurt
A letter printed in the Leicester Mercury newspaper said: "He was a freak of nature... our poor city has become ugly enough without a statue of this poor man being displayed."
The appeal's Facebook page has been challenged, with one person questioning whether Merrick deserves a statue, saying: "He had a rare condition and apparently that makes him somewhat amazing."
Other comments from people called it a "freak show", or said "let's have another [statue] of a bear dancing on hot coals", and "disfigurement should not be celebrated".
Despite only raising a fraction of the cost, Mrs Vigor-Mungovin said she was still trying to realise her dream, with fundraising events, new designs for the statue and a possible exhibition of items related to Merrick.
"I think the maquette [miniature design] will put a lot of people's mind at rest," she said.
"And I am hoping to get items from the Royal London Hospital for the exhibition, maybe even the full-size copy of his skeleton."
It is not the only statue of a notable Leicester figure being planned for the city - and others appear to have been more warmly received.
And last April then-MP Keith Vaz said a statue should be built in memory of Leicester City chairman Vichai Srivaddhanaprabha, who died in a helicopter crash outside the club's stadium in 2018. The design Mr Vaz suggested was of two elephants, with the animal being a symbol of good luck in Thailand.
Artist Ms Lapper, who was born without arms and with shortened legs, became famous when a statue of her, naked and pregnant, was displayed on Trafalgar Square's fourth plinth.
"Attitudes have changed since then, but not hugely," she said. "I remember sitting below my statue and seeing children being hurried away, people muttering it was 'inappropriate' but it started a conversation which is still going on today.
"I can't imagine how hard it must have been being disabled in the Victorian era, it was hard enough for me being born in the 60s.
"It would be great to see a statue of Joseph, especially if it highlights his courage getting himself out of the workhouse.
"People still feel uncomfortable around disability but if that's all that stopping this, then I say 'Tough, get over it'."
Mrs Vigor-Mungovin said: "People's ideas of him are dominated by the film from the 1980s, they want to feel sorry for him.
"But the real story is he had quite a good life, all things considered.
"He took control, he used his condition to his advantage, it's a powerful story."