At least 1,700 patients may have been harmed by a 'colossal' blunder that meant thousands of patient records were left to pile up in a warehouse.
The number at risk is likely to rise as only two thirds of the 700,000 notes found had been checked, officials said.
Cancer test results and child protection notes were among the documents that were missing in England.
The National Audit Office also said there were questions to answer about the handling of the incident.
Its review of the issue looked at the role of the government and the company responsible for the mix-up, which is part-owned by the Department of Health.
The company, NHS Shared Business Services (SBS), was employed in the East Midlands, South West and north-east London to redirect mail for the health service.
It was meant to pass on documents that had either been incorrectly addressed or needed re-routing because the patient had moved to a new GP surgery.
But between 2011 and 2016 a backlog of 709,000 pieces of correspondence piled up in a NHS SBS warehouse.
The issue came to light in February after the Guardian newspaper reported it.
Now the NAO has reviewed what exactly happened and found:
- The company had become aware of a risk to patients in January 2014, but senior managers had not developed a plan to deal with it or tell the government or NHS England for another two years
- A label with "clinical notes" on it had been removed from the room where the files were stored. A manager had apparently said: "You don't want to advertise what's in that room"
- In August 2015, a member of staff raised concerns the records were being destroyed
- NHS SBS finally told NHS England and Department of Health of the problem in March 2016, but neither Parliament nor the public were told
- The episode suggested there had been a conflict of interest between the health secretary's responsibility for the health service as a whole and his department's position as a shareholder in NHS SBS
- NHS England said the company had been "obstructive and unhelpful" when it had tried to investigate issue
The report by the NAO found the cost of dealing with the incident was likely to be in the region of at least £6.6m.
A spokeswoman for NHS SBS acknowledged there had been "failings".
She added: "We regret this situation and have co-operated fully with the NAO in its investigation."
A Department of Health spokeswoman said it was committed to being transparent over the handling of the issue and was working to make sure this did not happen again.
It said it was given advice not to raise the alarm publicly until it had a better understanding of the problem, concerns about patient safety would always outweigh its role as a shareholder in the company and as yet there had still been no proof of harm to patients.
Individual investigations - overseen by NHS England - are taking place into the 1,788 cases of potential harm identified by GPs who have reviewed the missing notes.
On top of that over 200,000 records have still to be reviewed by GPs in the first place to determine if there was a potential for harm to have happened.
All investigations are expected to be completed by the end of the year.
Dr Richard Vautrey, of the British Medical Association, said the "disastrous" situation should never happen again.
"The handling and transfer of clinical correspondence is a crucial part of how general practice operates, and it's essential that important information reaches GPs as soon as possible so that they can provide the best possible care to their patients."
Liberal Democrat leader Tim Farron described it as "colossal blunder".
And shadow health secretary Jonathan Ashworth said the whole episode was a "scandal" that ministers needed to answer for.
"This is a staggering catalogue of mistakes on this government's watch," he added.