"More than damaging to boxing, it's damaging to their legacy," five-weight world champion Leonard told BBC Sport.
"This is for history, for people to say 'I remember when'. It's beyond money."
Plans for a fight between Mayweather, the American who has won world titles at five weights between super-featherweight and light-middleweight, and Philippine Pacquiao, a seven-weight world champion, fell through in 2010 when Pacquiao rejected his opponent's demand for random drug-testing in the build-up to the bout.
There have been repeated attempts to renegotiate the fight in the intervening months, but to no avail.
BEN DIRS' BLOG
"When I decided to come back, everyone thought I was crazy. And for a while, I thought I was, too. I'd had one fight in five years and I was coming back to fight Marvin Hagler, a man who hadn't lost a fight since 1976, a man who had beaten everybody from Duran to Hearns."
Meanwhile, Hearns's trainer Emanuel Steward told BBC Sport the failure of Mayweather and Pacquiao to meet in the ring made him fear for the future of the sport.
Steward, who trains world heavyweight champion Wladimir Klitschko, told BBC Sport: "There's only one major fight the public is interested in - two little fellas, Floyd Mayweather and Manny Pacquiao, who most likely will never fight.
"Boxing's not in good shape. It's more of a business now and it's hurting boxing. We need to return to an era when the best fight the best and it wasn't all about money and egos.
"Unless some drastic changes come about, I don't like the way things are looking for boxing. There's not one dream heavyweight fight - you've got Wladimir Klitschko and Vitali Klitschko and nobody else. That's not good.
"What made that [1980s] era great was that they fought each other. They could have played politics but they were true warriors."
Steward, who also trained Britain's Lennox Lewis, added that the first fight between Leonard and Hearns in 1981 took less than a day to negotiate, from start to finish.
Steward said: "I could see something building up, so finally I called up [Leonard's manager] Mike Trainer and said 'Mike, it looks like the public wants to see Ray and Tommy fight and he said 'why don't we meet today?'
"So I jumped on a plane and we met at an airport coffee shop. Mike Trainer said 'what are you looking for for Tommy?' I said, 'I want $5m for Tommy'. He said 'have you lost your mind? How much do you think Ray should get?' And I said $8m.
"They were all shocked, they thought I was going to ask for more for Tommy. They stepped out for a minute, came back and said 'pal, you've got a deal'. I shook hands and flew back to Detroit on the exact same plane that I came on - that's how quick the deal was made.
"One of the biggest fights in history was made on a quick handshake in about 30 minutes. There were no egos involved. We knew the public wanted it.
"The money is there [for Mayweather-Pacquiao], the two guys would make a minimum of $50m apiece, and it still can't be made because of egos. Times have changed and it really frustrates me because I remember how easy it was for me to make the fight with Ray."
Sugar Ray Leonard boxing special, BBC Radio 5 live, Tuesday 3 April, 1930 BST
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.