Asylum seekers were housed in cramped and filthy conditions at a military barracks, inspectors have said.
Some of the most vulnerable people were living in a "decrepit" block unfit for habitation at Kent's Napier Barracks.
The Independent Chief Inspector of Borders and Immigration and Her Majesty's Inspectorate of Prisons also visited Penally Camp in Pembrokeshire. They said both sites were "run-down".
The Home Office said it had instructed service providers to make improvements.
Labour has said the findings are "utterly unbelievable", and Plaid Cymru called for Home Secretary Priti Patel to "consider her position".
Inspectors said both sites - which witnessed protests over living conditions in January - were "run-down and unsuitable", but particularly criticised Napier.
They said given the cramped, communal conditions, "once one person was infected [with Covid] a large-scale outbreak was virtually inevitable" at the site.
In January, 178 people tested positive for the virus at Napier.
Many men said they were depressed, and a survey found a third of respondents at both sites had mental health problems, while a third at Napier had felt suicidal.
Inspectors said contractors were given less than two weeks to make both sites operational, and local stakeholders, including healthcare, were not consulted and were given insufficient time to prepare.
They listed concerns about the asylum process, including that many residents were told they would be there for weeks, but had been there several months - a major cause of distress that left them not trusting anything they were told.
Ms Patel, who has repeatedly defended the use of Napier, last month said: "This site has previously accommodated our brave soldiers and army personnel. It is an insult to say that it is not good enough for these individuals."
Shadow immigration minister, Holly Lynch, said: "The Home Secretary has recklessly put the lives of all those staying at the sites, those working there, and the surrounding communities, at risk during a pandemic."
Plaid Cymru's Westminster leader Liz Saville Roberts said Ms Patel "must consider her position" in the wake of the report, which highlighted "fundamental failures of leadership and planning by the Home Office".
A spokeswoman for the Home Office said: "During these unprecedented times we have met our statutory duty to provide asylum seekers, who would otherwise be destitute, with suitable accommodation and three meals a day all paid for by the British taxpayer.
"We expect the highest possible standards from our service providers and have instructed them to make improvements at the site."
Following inspections in February and a follow-up visit to Napier on Thursday, inspectors will now produce a detailed report that will be submitted to Ms Patel.
The findings were released on the same day that 115 people crossed the English Channel in seven boats in continued attempts to reach the UK, which the government has said are facilitated by criminals.
Last year, nearly 8,500 people attempted the journey.
The Home Office has said people should claim asylum in the first safe country they reach rather than making the crossings, and has said illegal migrants are being returned to safe countries.