MI5 has secretly been collecting vast amounts of data about UK phone calls to search for terrorist connections.
The programme has been running for 10 years under a law described as "vague" by the government's terror watchdog.
It emerged as Home Secretary Theresa May unveiled a draft bill governing spying on communications by the authorities.
If it becomes law, the internet activity of everyone in Britain will be held for a year by service providers.
Police and intelligence officers will then be able to see the names of sites suspected criminals have visited, without a warrant.
Mrs May told MPs the proposed powers were needed to fight crime and terrorism but civil liberties campaigners warned it represented to a "breathtaking" attack on the internet security of everyone living in the UK.
The draft bill aims to give stronger legal cover to the activities of MI5, MI6 and the police and introduce judicial oversight of spying operations.
It confirmed that Britain's secret listening post GCHQ has been intercepting internet messages flowing through Britain in bulk, as revealed by US whistleblower Edward Snowden, "to acquire the communications of terrorists and serious criminals that would not otherwise be available".
It also revealed that the UK security services have been allowed to collect large amounts of data on phone calls "to identify subjects of interest within the UK and overseas", provided they comply with certain safeguards, set out in a supporting document also published on Wednesday.
The draft bill aims to tighten up these safeguards and put the bulk collection of data on a firmer legal footing. Taken together with the other measures, the home secretary said the bill would give the security services a "licence to operate".
Analysis by BBC Security Correspondent Gordon Corera
While GCHQ's programmes were exposed by Snowden, this one by MI5 remained secret.
And in a way that became increasingly awkward for the security service as the drive towards being more open about capabilities picked up pace in the wake of the report by David Anderson, the independent reviewer of terrorism legislation, earlier in the year.
There were hints about the capability in the speech by MI5 boss Andrew Parker the week before the draft Investigatory Powers Bill was published, when he talked about how "accessing data quickly, reliably and at scale is as fundamental to our work…..without communications data for example we could not have detected and disrupted numerous plots over the last decade."
He, like the home secretary, claimed that bulk communications data was used to "identify, at speed, links between the individuals plotting to bomb the London Stock Exchange in 2010".
Now - along with other capabilities - the bulk data programme is out in the public and up for debate.
In her Commons statement, Mrs May referred to the 1984 Telecommunications Act, under which she said successive governments had allowed security services to access data from communications companies.
The data involved the bulk records of phone calls - not what was said but the fact that there was contact - with companies required to hand over domestic phone records.
BBC security correspondent Gordon Corera said the programme, which sources said was used to track terrorists and save lives, was "so secret that few even in MI5 knew about it, let alone the public".
'Not outside the law'
The government's independent reviewer of terrorism legislation, David Anderson QC, told the BBC the legislation used to authorise the collection was "so vague that anything could be done under it".
He added: "It wasn't illegal in the sense that it was outside the law, it was just that the law was so broad and the information was so slight that nobody knew it was happening".
Mr Anderson has called for a "comprehensive" new law governing surveillance, which the government has produced with the wide-ranging draft Investigatory Powers Bill.
The draft bill's measures include:
- Allowing the security services to hack into phones and computers around the world in the interests of national security
- Giving a panel of judges the power to block spying operations authorised by the home secretary
- A new criminal offence of "knowingly or recklessly obtaining communications data from a telecommunications operator without lawful authority", carrying a prison sentence of up to two years
- Local councils to retain some investigatory powers, such as surveillance of benefit cheats, but they will not be able to access online data stored by internet firms
- The Wilson doctrine - preventing surveillance of Parliamentarians' communications - to be written into law
- Police will not be able to access journalistic sources without the authorisation of a judge
- A legal duty on British companies to help law enforcement agencies hack devices to acquire information if it is reasonably practical to do so
- Former Appeal Court judge Sir Stanley Burnton is appointed as the new interception of communications commissioner
Mrs May told MPs the draft bill was a "significant departure" from previous plans, dubbed the "snoopers' charter" by critics, which were blocked by the Lib Dems, and will "provide some of the strongest protections and safeguards anywhere in the democratic world and an approach that sets new standards for openness, transparency and oversight".
The proposed legislation will be consulted on before a bill is formally introduced to Parliament in the New Year, Mrs May said. It will then have to pass votes in both houses of Parliament.
Labour's shadow home secretary Andy Burnham backed the draft bill, saying it was "neither a snoopers' charter nor a plan for mass surveillance".