The International Olympic Committee has warned it will move the Tokyo 2020 golf event from its current venue if it does not admit women.
The Kasumigaseki Country Club does not allow women to become full members or play on Sundays.
"I respect it's a private club but our position is clear. We will only go to a club that has non-discrimination," said IOC vice-president John Coates.
"At some point there has to be a cut-off."
However, Coates added that he did not expect to have to find another host for the event.
"It's possible to go elsewhere but I think this is going to work out," he added.
"My understanding is as recently as this week there have been more discussions with the club, that it's heading in the right direction for them to have a non-discriminatory membership.
"It would appear that we should be able to have this resolved by the end of June."
In February, club chairman Kiichi Kimura described the controversy as "annoying" after initial internal discussions had not resulted in any decision on Kasumigaseki's membership policy.
"We are baffled," he said.
"We agreed to host at their request, but we never made a bid."
World number one and Rio 2016 silver medallist Lydia Ko has said that she wants to see the bar to women at the club lifted, while the Japan Golf Council - a group aimed at modernising the game - has sent the IOC a letter recommending an alternative course.
Rio 2016 was the first time golf event had been part of the Olympic programme since 1904. South Korea's Inbee Park won the women's title, with Great Britain's Justin Rose securing the men's.