More than 40 faith group leaders have written to the government opposing plans to build temporary accommodation to house about 200 asylum seekers.
The Home Office wants to build the units beside Yarl's Wood immigration removal centre in Bedfordshire.
The Bedfordshire-based groups object to the location and doubt the type of accommodation can meet residents' "health and mental well-being" needs.
The Home Office said they would be "Covid-secure, safe [and] habitable".
In December, the government was housing 61,327 asylum seekers compared to about 47,000 a year earlier, according to the Home Office.
It said it needs to find more accommodation in order to meet its legal obligations.
A Home Office spokeswoman said: "All accommodation will be Covid-secure, safe, habitable, fit for purpose and correctly equipped in line with existing standards and contractual requirements.
"The safety and security of the local community, asylum seekers, staff and visitors are of paramount importance."
Objections from the faith and community leaders, who come from Christian, Sikh, Muslim, Hindu and Baha'i organisations, include its proximity to the detention centre, the "significant difficulties and trauma" faced by many of the intended residents, the type of accommodation and Yarl's Woods' "distance from urban centres".
Reverend Luke Larner, from Red Letter Christians UK in Bedford, said: "We would like a better alternative that is beneficial for the wellbeing of these people."
Yarl's Wood, which opened in 2001, can hold about 400 detainees.
The majority were women until August, when the last few female detainees left the site as a result of the Covid-19 pandemic.
The centre has since been used to house dozens of Channel migrants.
Bedford resident Rosie Newbigging is also opposing the plans to "place people seeking asylum in portacabins on wasteland".
Earlier this month, her solicitor sent letters to the Home Office and Bedford Borough Council threatening legal action.