Swimming has seen a 245,000 fall in participation numbers over the last year, according to Sport England.
A total of 2.69m people swam at least once a week for 30 minutes between October 2013 and October 2014.
The number of people participating regularly in sport is also down, by 125,000 to 15.6m, while 121,700 fewer disabled people play sport regularly.
"I am disappointed with these figures," said Sport England chief executive Jennie Price.
Swimming remains the country's most popular sport but Price added she was "very concerned" about the drop in numbers.
The outcome of Sport England's latest survey means the Amateur Swimming Association must wait until March to find out if its funding is affected.
|Sport England chief Jennie Price|
|"Swimming has lagged behind running and the gym in terms of offering an attractive, modern experience to people who want to play sport and exercise. That has to change and to change quickly."|
ASA boss Edward Lord said: "There is clearly a major issue here and we will be putting all our resources into finding out both the cause of the fall in participation numbers and in identifying how we can move quickly with our partners to reverse that decline."
Team sports - including football, cricket, netball and rugby union - recorded an increase in participation.
Athletics, canoeing, mountaineering, taekwondo and fencing also had a rise in the number of people taking part.
|Who's taking part in what|
|Oct 2012-Oct 2013||Oct 2013-Oct 2014||Change|
|Source: Sport England's Active People Survey|
The number of 14-25-year-olds in England playing sport has increased by 55,900 to 4.72m, while the overall number of people taking part in regular activity remains 1.6m higher than it was when London won the right to host the 2012 Olympics in 2005.
Sport England also found that 1.75m fewer women than men take part in regular sport.
Sports minister Helen Grant said she was "very concerned" about the overall fall in sports participation over the last 12 months.
She also said that sports bodies needed to do more to involve women and ethnic minority groups.
|Table Tennis England chief Sara Sutcliffe|
|"Changes are not always easy but in this case our major partner and key funder, Sport England, was right to put pressure on us to change."|
"I expect Sport England to take tough decisions and redirect funds from governing bodies that are failing to projects and organisations that will deliver," said Grant.
Clive Efford, Labour's Shadow Minister for Sport, was not impressed with the latest figures.
"The Government were handed a golden sporting legacy which they have completely failed to capitalise on," he said.
Three sports will get longer-term investment as a reward for their progress over the last 12 months.
The Lawn Tennis Association, Table Tennis England and British Fencing will now be guaranteed funding up until March 2017.
Like swimming, Basketball England will have to wait until March to find out if it has retained its funding.