Asylum seekers housed in a former army base have protested at conditions at the "institutional" ex-military site.
Residents at the Napier Barracks in Folkestone are suffering high levels of trauma, according to the Helen Bamber Foundation, which works with refugees.
Dr Jill O'Leary, a GP who works with the charity, said asylum housing was supposed to be in the community and institutional sites were "inhumane".
The Home Office said residents were "free to come and go" from the centre.
Chris Philp, Immigration Compliance Minister, said: "We provide asylum seekers with safe, warm, Covid secure suitable accommodation where they receive three meals a day whilst their claims are being processed.
"They are not detained and are free to come and go in line with Covid-19 restrictions. We have a robust complaints process where the people who we support or people representing them can raise concerns."
The Helen Bamber Foundation said some started refusing food on 8 January, while others slept outdoors in freezing temperatures, rather than sleep inside.
More than 100 asylum seekers walked out of the barracks on Tuesday for about an hour, the BBC was told.
In a statement, a Sudanese asylum seeker said he had slept outside for five days.
"Thirty-four of us share one shower and one toilet. There is no privacy. At night no-one can sleep because there is too much noise. The situation in unbearable," he said.
"It's a military setup here. It's like being in prison. You can't do anything without someone knowing. Everything you do is watched."
Clare Mosely, from Care4Calais, another charity supporting refugees, said: "It's very cold, we've seen photographs of bathrooms with no soap. They are also worried about Covid and believe they aren't getting the same protection as everybody else.
"What they would really like is for their asylum claims to be processed."
Kent Police said officers were called to the barracks at about midday and remained there to support the Home Office and ensure public safety.