The Gay Football Supporters' Network has told Newsbeat some Premier League clubs aren't doing enough for gay fans.
The campaigners say getting adequate help setting up official supporters' groups for lesbian, gay, bisexual and transgender fans is too difficult.
"I definitely think all Premier League clubs should have an official one," explained GFSN chairman Ed Connell.
The organisation, which also runs a national LGBT league, says such groups can help tackle homophobia.
"There are some clubs that still seem to think, remarkably as it may seem that there isn't a problem with homophobia within football," added Connell.
"They think because there is no homophobic chanting within their grounds there's no need to actively help an LGBT fan group which is disappointing."
There are hundreds of different supporters' groups in the UK and they usually give fans from the same club, who live in a certain area, the chance to meet up and socialise.
Few of them are specifically for the LGBT community, though.
In 2013 Arsenal became the first major English club to recognise such a group - the likes of Manchester City and Everton followed.
But Newsbeat has found not all clubs have an LGBT group.
Apart from Arsenal, we asked the other 19 Premier League clubs if they had an official one connected with the club.
Of the 11 teams that got back to us four chose not to answer the question, five said they did. Stoke City and Sunderland confirmed they didn't have a specific group at the moment.
Arsenal's Gay Gooners say the group is important because they work with the club to tackle homophobia and even have a rainbow banner at the Emirates.
"It's just a social group for like-minded supporters to meet up and have a few drinks," explained Steve, a member of the Gay Gooners.
In most cases supporters' groups of any kind are formed by fans approaching a club and demonstrating there's a need for one.
Chris Paouros from Pride in Football said that can be "tricky" for gay supporters.
"We know that LGBT fans sometimes don't go to football because it doesn't feel like a safe place to be," she said.
A recent Out on the Fields survey, taken by more than 9,000 people, claims 77% of UK participants had witnessed or experienced homophobia in sport.
Chris says it's important that clubs are "engaged".
It's that kind of attitude which led to an LGBT supporters' group being set up at Tottenham.
"They (Tottenham) used all their resources to find the supporters, to pull them together," explained Simon Gray, who helped set up the Proud Lilywhites.
He said that was important because "it was really empowering to see the club reach out and try to find people".
Fans at last weekend's Community Shield could see the benefits of helping gay supporters but some warned of "categorising fans".
"I feel we don't need one because we are all just one club but if people do want the support we should have one," said Chelsea fan Luke Viney.
Arsenal supporter Jenny Spicer said: "I don't think you'll get all 20 teams all on board with this."
Her friend, Ryan Gregory, added: "As a gay supporter it's nice to go to a game and know you're supported."
For GFSN chairman Ed Connell, the importance of such groups is a significant tool in educating fans.
"To see the Gay Gooners displaying their banner inside the Emirates is the most amazing thing for an LGBT fan who's been attending football for years but felt like the silent minority within the ground."