Olympic chiefs will look at the number of top golfers not playing at Rio this summer when deciding if the sport should be at future Games.
The world's top four are among more than 20 to have withdrawn.
International Olympic Committee president Thomas Bach said this would be considered when the line-up for Tokyo 2020 is decided next year.
"One of the main categories is the question of participation by the best players," he said.
Critics say golf should not keep its Olympic place because of the lack of support shown by leading players. Squash was one of the sports that lost out to golf in the bidding for an Olympic place, and English player Laura Massaro said the lack of appreciation for the Olympics by some golfers was "unacceptable".
Golf is returning to the Olympics this summer after a 112-year absence, following a successful bid that included leading golf personalities helping to make the sport's case.
But some of those golfers have withdrawn because of the threat of Zika - a mosquito-borne virus linked to brain defects in newborn babies. That is despite World Health Organisation advice saying the risk to athletes at the Games was low. By contrast, only one player has pulled out of the women's tournament.
Other male players, though, said the Olympics held no allure for professional golfers who would rather compete for the sport's four major trophies - such as the Open, which starts at Royal Troon on Thursday.
Ahead of the Open, world number four Rory McIlroy, who had been expected to play for Ireland, emphasised that point. He even went as far to suggest he would not even watch golf on television at the Olympics, instead choosing sports "that matter" such as track and field.
Speaking on Wednesday, Bach said: "We have to respect the individual decisions, even if they are going contrary to the World Health Organisation recommendations.
"On the other hand, we are aware of discussion in the golf community that there are obviously very different reasons for not going to Rio, not related to Zika."
We've launched a new BBC Sport newsletter, bringing all the best stories, features and video right to your inbox. You can sign up here.