The UK's intelligence services are facing an "intense" challenge from terrorism, the head of MI5 has warned.
Andrew Parker said there was currently "more terrorist activity coming at us, more quickly" and that it can also be "harder to detect".
The UK has suffered five terror attacks this year, and he said MI5 staff had been "deeply affected" by them.
He added that more than 130 Britons who travelled to Iraq and Syria to fight with so-called Islamic State had died.
MI5 was running 500 live operations involving 3,000 individuals involved in extremist activity in some way, he said.
Speaking in London, Mr Parker said the tempo of counter-terrorism operations was the highest he had seen in his 34-year career at MI5.
Twenty attacks had been foiled in the last four years, including seven in the last seven months, he said - all related to what he called Islamist extremism.
The five attacks that got through this year included a suicide bomb attack after an Ariana Grande concert at Manchester Arena in May, killing 22.
Five people were also killed in April during an attack near the Houses of Parliament, while eight people were killed when three attackers drove a van into pedestrians on London Bridge and launched a knife attack in Borough Market.
A man then drove a van into a crowd of worshippers near a mosque in north London in June, while a homemade bomb partially exploded in tube train at Parsons Green station last month, injuring 30 people.
In some cases, individuals like Khuram Butt - who was behind the London Bridge attack - were well known to MI5 and had been under investigation by the security services.
Mr Parker was asked what was the point of MI5 surveillance when someone who had made "no secret of his affiliations with jihadist extremism" had then been allowed to go on to launch a deadly attack.
He said the risk from each individual was assessed on a "daily and weekly basis" and then prioritised "accordingly".
"One of the main challenges we've got is that we only ever have fragments of information, and we have to try to assemble a picture of what might happen, based on those fragments."
He said the likelihood was that when an attacked happened, it would be carried out by someone "that we know or have known" - otherwise it would mean they had been looking "in completely the wrong place".
And he said staff at MI5 were deeply affected on a "personal and professional" level when they did happen.
"They are constantly making tough professional judgements based on fragments of intelligence; pinpricks of light against a dark and shifting canvas."
'Not the enemy'
Mr Parker said they were trying to "squeeze every drop of learning" from recent incidents.
In the wake of attacks in the UK, there had been some, including some in the Home Office, who questioned whether the counter-terrorist machine - featuring all three intelligence agencies and the police, and with MI5 at its heart - was functioning as effectively as previously thought.
However, there was no indication of a fundamental change in direction in his remarks, with a focus on the scale of the threat making stopping all plots impossible.
"We have to be careful that we do not find ourselves held to some kind of perfect standard of 100%, because that is not achievable," he said.
"Attacks can sometimes accelerate from inception through planning to action in just a handful of days.
"This pace, together with the way extremists can exploit safe spaces online, can make threats harder to detect and give us a smaller window to intervene."
He renewed the call for more co-operation from technology companies.
Technology was "not the enemy," he added, but said companies had a responsibility to deal with the side effects and "dark edges" created by the products they produced.
In particular, he pointed to online purchasing of goods - such as chemicals - as well as the presence of extremist content on social media and encrypted communications.
He said more than 800 individuals had left the UK for Syria and Iraq.
Some had then returned, often many years ago, and had been subject to risk assessment. Mr Parker revealed at least 130 had been killed in conflict.
Fewer than expected had returned recently, he said, adding that those who were still in Syria and Iraq may not now attempt to come back because they knew they might be arrested.
Mr Parker stressed that international co-operation remained vital and revealed there was a joint operational centre for counter-terrorism based in the Netherlands, where security service officers from a range of countries worked together and shared data.
This had led to 12 arrests in Europe, he added.
In terms of state threats, Mr Parker said the range of clandestine activity conducted by foreign states - including Russia - went from aggressive cyber-attack, through to traditional espionage and the risk of assassination of individuals.
However, he said the UK had strong defences against such activity.