People from 10 homes have been ordered to leave immediately due to an imminent risk of a landslip which could cause their houses to collapse.
A series of landslips on a hillside behind the properties in the Swansea Valley has led to the warning.
Neath Port Talbot council said lives could be at risk unless the homes on Cyfyng Road, Ystalafera were evacuated.
But some residents have said they were not prepared to leave.
There have been two further landslides this year and there are fears that another could cause the houses to collapse.
Council bosses have said there was no immediate solution to the issue.
Deputy leader Anthony Taylor said the council was trying to re-house those affected.
Resident Paul Harris said there had been two landslides from the hillside behind his rented home earlier this year, but a survey had found no damage to his house.
He said: "Two days ago a letter arrived in the post telling me I had to move out immediately because there was an imminent risk of the house actually falling down."
Despite that warning, he said he felt safe and would not leave because he had only been offered alternative bed and breakfast accommodation but could not take his two dogs with him.
He said: "Maybe there is a real risk, I don't know."
"I think the council need to decide on a definite plan of action. They seem to be not really giving residents the information that's being requested," he added.
"If they knew that it was likely that they would have to do this, why wasn't accommodation organised earlier, why now?"
The homes affected are a mix of privately owned and rented accommodation and Mr Taylor told BBC's Good Morning Wales programme there had been problems in the area since the 1940s, due to the geology of the land.
He said further ground movement in February and June had identified new issues and engineers advised further slips were possible in bad weather.
Mr Taylor said it could spark rising ground water which could cause structural or sewerage problems in the 10 homes identified as at risk.
He admitted the situation could "go on for some time" and when asked if it could be indefinite, he replied "yes".
A council advice centre and hotline have been set up to support residents and Mr Taylor said its housing team was working with social landlords and other providers to find accommodation for those who needed it.
He said: "We have great sympathy for those residents involved and it's a very difficult time for them.
"We are working with the individual families, because each of them has their own set of circumstances, and we are working to find accommodation as close as possible to where they are."