The Government is "really concerned" about the rising cost of tickets, and has asked for football clubs to review their pricing.
Minister for Sport Helen Grant told the BBC: "I feel clubs really must not take their fans for granted."
The average price of the cheapest tickets across English football has risen at almost twice the rate of the cost of living since 2011.
BBC Sport's Price of Football study analysed prices at 207 clubs.
The average price of the cheapest match-day ticket from the Premier League to League Two in England is now £21.49.
That has increased 13% since 2011, compared to a 6.8% rise in the cost of living. Year-on-year it is up 4.4%, more than treble the 1.2% rate of inflation.
"I can see why fans are cross. I'm cross," said Grant. "Fans are the lifeblood of the game, without the fans we won't have football the way we know it.
"To take a family of four to a Premier League match now you're talking about £130, and that's before petrol, parking, a programme, hot dogs, burgers and a drink.
"That is just not affordable for most families so clubs really do need to look at this."
Arsenal's Supporters' Trust is expected to demand a price freeze when it addresses the club at its annual general meeting on Thursday, after the study found the Gunners have the most expensive match-day ticket in the Premier League.
A statement read: "The ticket price increase this season has created bad feeling amongst the fanbase, made worse when we see the club sitting on such a large unused cash pile.
"Will the board consider the difficulty fans have with keeping up with ever increasing costs of attending football and confirm there will be no increase next year?"
|Price of Football 2014 - the key findings|
|Arsenal have the most expensive match-day ticket in the Premier League at £97|
|Manchester City have the league's cheapest season ticket at £299. That's cheaper than at 15 Championship clubs, 10 clubs in League One, four in League Two and even one in the Conference.|
|Charlton's £150 season ticket is the cheapest in England's top four divisions. However, Barcelona charge about £103 for their lowest-priced season ticket.|
|In Scotland's Premiership, the average price for the cheapest day at a match, including a ticket, pie, cup of tea and a programme, increased by just 61p to £26.95 from last season.|
This year's Price of Football study found the cheapest Premier League season ticket is £299 at Manchester City.
Danny Wilson, the club's director of sales, said: "When you scratch down deeper into the work we're doing, it's about creating a better experience for our supporters.
"We want to make sure coming down to the stadium is a far broader and more engaging experience than just the 90 minutes of football.
"Our long-term goal is to keep growing our fanbase and keep filling the stadium week-in, week-out."
The Premier League has pointed to packed-out grounds as a sign it is getting the pricing right.
"The attendances so far this season are very encouraging, with more than 95% of seats sold and average crowds tracking with last season's, which were the highest in English top-flight football since 1949-50," said Cathy Long, the Premier League's head of supporter services.
Football League chief executive Shaun Harvey said the Price of Football study showed clubs are rewarding fans for their loyalty and financial commitment with better-value season tickets, which has led to prices falling in the period since 2005.
Crowds also increased by 136,000 across the Football League last season.
Critics of the price hikes said clubs had lost touch with fans and argued the £5.5bn windfall from the current television rights deal should have resulted in a drop in ticket prices for supporters.
Justin King, former chief executive of British supermarket Sainsbury's, feels clubs could fail if they do not meet the needs of supporters.
"Any business that thinks it can simply rely upon the loyalty of its customers, regardless of how they treat them, in the end will fail," King, a Manchester United supporter, told the BBC.
"I would be asking clubs: 'Are your fans happier today than they were five years ago with the experience that they get, the value for money that they feel they're getting?'"
The BBC's economics editor Robert Peston pointed to factors such as rising players' wages in the Premier League and the fact a majority of clubs operate at a loss.
"Yes, clubs do need to take a commercial decision, they set the ticket price, but there's got to be reasonableness," Grant said.
"I have two lads. They both love football and now and again we went to see a match but it was very, very expensive and it's the affordability.
"Football can be good family entertainment, it's our national game, but you need to be able to get through the gates."
Some clubs are adopting discount initiatives to help cash-strapped supporters.
Manchester City midfielder Frank Lampard illustrated some of those schemes and backed the Price of Football study for unearthing its findings.
"I have been at Chelsea and Manchester City and they are making big strides, they are trying to freeze prices and encourage people to come to games," he said.
"They reduce prices for certain away games as well as other incentives and they need to do more of that."
Grant backed up those schemes, but called on clubs to do more.
"Although I'm concerned about this survey, there's some good practise happening," she added.
"I know, for example, West Ham run their 'kids for a quid' promotion. This is the sort of thing I want to see more of."
Grant revealed the government has set up a panel of senior football administrators to give supporters the opportunity for their concerns to be raised.
"We've set up an Expert Group. You're going to hear more about that soon - we're about to launch it - and this group will look at very important issues such as... pricing, club ownership and debt, seating.
"It will give fans the profile and platform to air their views."