The Bishop of Bristol said comments in a sermon which dismissed Edward Colston's links to the slave trade as "speculation" were "misinterpreted".
Remarks about the 17th Century slave trader were made by the Right Reverend Michael Hill at Bristol Cathedral.
He was reported as saying, to a private audience, that "there may be still some speculation around his business roots".
Bishop Hill said he was referring to the "speculation around the morality of Colston's business dealings".
Edward Colston, an official of the slave trading Royal African Company, made a significant proportion of his money directly or indirectly from the trade.
He also donated considerable sums to charitable causes across the city, resulting in an ongoing controversy about how he is remembered.
On Monday, Bishop Hill was addressing a private audience at the annual Merchant Venturers' Charter Day service at the cathedral.
According to Tony Gosling, who was in the congregation, the bishop said he thought Edward Colston had "lived a life of significance" and there "may be still some speculation on some of the circumstances around his business roots right here [in Bristol]".
"What I heard the bishop say was: 'It is speculation'" he said.
'Misinterpreted by some'
But Bishop Hill said he was "against all forms of slavery" and was "sad" his message had been "misinterpreted by some".
"The speculation, I was trying to refer to was there are some people who see Colston as completely tainted by his past and there are others who see his efforts as philanthropic," he said.
"I was in front of a congregation that had three schools in it, and two of them had directly benefited from the benefaction of Colston.
"Colston, despite his ill-gotten gains, did some things that were significant and the children and elderly in our city have benefitted from his benefaction but at the same time it does not in any way justify his attachment to the Royal African Company."