A minority at Yorkshire believe "nothing was wrong" at the club, despite sweeping reforms amid the racism scandal, says chair Lord Patel.
The club was heavily criticised after a report found Rafiq was a victim of "racial harassment and bullying".
Patel said: "There is a very small minority who believe nothing was wrong here and wish to return to those days.
"That is my fear, that those people continue to believe that."
Patel, who was appointed in November when the scandal engulfed cricket, said "85% of members" voted for the reforms.
The changes meant Yorkshire could host international matches again, having met criteria set out by the England and Wales Cricket Board (ECB). In November, the ECB suspended the club's right to stage England games in response to the outcome of the report.
Before the emergency meeting of members, Patel said a group of individuals was "actively seeking to delay and derail" reform at Yorkshire.
Speaking to the BBC during Yorkshire's County Championship match against Lancashire at Headingley on Friday, Patel added: "I have met thousands of people here who are genuine, good people who want to do the right thing.
"You have to believe the majority of people here want to do the right thing and go in the right direction."
Rafiq, 31, was born in Pakistan and moved to England aged 10 before playing for Yorkshire between 2008 and 2018.
However, in September 2020, following an initial interview with Wisden, Rafiq told ESPN Cricinfo that "institutional racism" encountered while he was at the club left him close to taking his own life.
He told BBC Sport he dreaded "every second" of his career and a team-mate used a racially offensive term linked to his Pakistani heritage.
Yorkshire initially said no-one would be disciplined.
Director of cricket Martyn Moxon, head coach Andrew Gale and all members of the coaching staff were among 16 people to leave Yorkshire under the new regime in December.
Although the club is now able to host men's England matches against New Zealand and South Africa this summer, it is still awaiting the results of a separate ECB regulatory investigation.