English football is "dysfunctional" in the way it tackles racism, says Kick It Out chairman Lord Ouseley.
There have been several incidents of racist abuse involving supporters in domestic football this season.
Lord Ouseley, 73, who founded the anti-discrimination body in 1993, told the BBC: "Those with power do not exercise their responsibilities in the way that will help us deal with this problem.
"Instant action is what we're looking for - responsible leadership."
Premier League bosses met with the 20 top-flight chairmen on Thursday and said: "All clubs agreed that more needs to be done to ensure BAME [Black, Asian and minority ethnic] fans and players are supported, and that pathways are improved to encourage more people from different backgrounds to take up coaching, refereeing and administrative roles in the sport."
Lord Ouselely, who will step down from his role this year, said Kick it Out should have been included earlier in discussions.
He said: "The fact that the Premier League has not consulted with Kick It Out, where there is a considerable amount of expertise and experience to assist them, is contemptuous of the way in which football is.
"It's sad that the Premier League, who we do receive quite a lot of support from, does not value that in the way in which it is has moved this forward.
"Whether it sees doing this secretly and on its own to gather the acclaim it is maybe seeking because it is right up front with an initiative of its own, I think it is the wrong way to go about doing things and I hope they will see the error of their ways.
"The problem with football at the present time is it's very dysfunctional in the way it's spread across the whole of the game with the Football Association, the PFA, the Football League, the Premier League, the League Managers Association and so on."
'Leadership in football is not responsible as it should be'
Lord Ouseley highlighted the case of Manchester City and England forward Raheem Sterling being allegedly abused by Chelsea supporters in December.
After the incident, Sterling said newspapers helped to "fuel racism" by the ways in which they portray young black footballers.
"The question that had to be asked is why did the FA not see what was going on and take action on that day?" Lord Ouseley said.
"Where was his players union, the PFA, in saying something?
"Where was the Premier League? It was their match being broadcast across the whole world and this was happening, and it was left to Raheem Sterling to expose that, and that is where there is a problem. Leadership in football is not active and responsible as it should be."
The Football Association said it is investigating racist chants from a section of Millwall fans during the club's FA Cup victory over Everton last month.
West Ham have handed evidence to the Metropolitan Police after a video clip filmed by a supporter at London Stadium on Monday showed Liverpool's Egypt forward Mohamed Salah being subjected to abuse when taking a corner.
It follows a spate of problems in December, including a fan throwing a banana skin, Chelsea fans making anti-Semitic chants and a bottle striking Tottenham's Dele Alli.
There were also reports of homophobic chants at a Huddersfield-Brighton game.
Sports minister Mims Davies announced plans this week to hold an "urgent" meeting with football leaders to try to tackle racism and discrimination.
FA chairman Greg Clarke said: "We're coming up with a cohesive approach between stakeholders and between government on how we can improve it.
"Anybody who has a set of rules that thinks they can't be improved has to look hard at themselves, and we're working with everybody to try and find ways we can improve things."