Communities Secretary Eric Pickles has warned that an "influx" of Romanians and Bulgarians would add to the existing housing problems in the UK.
But he refused to give an estimate of the numbers of people who might move to Britain after getting the right to live and work in the UK in December.
"I don't think anybody knows," he told the BBC, adding that more work had to be done on drawing up a robust figure.
He said he did not want to start a "scare story" on the issue.
Temporary curbs were imposed on Romanians and Bulgarians in 2005 to protect the British labour market, but those curbs expire in December and under EU laws cannot be extended.
Mr Pickles was asked about the impact of a fresh wave of EU migrants by Andrew Neil on the BBC's Sunday Politics after a discussion about the low rate of housing growth.
Asked how many Romanians and Bulgarians he expected to move to the UK, he said: "The truth is I don't think anybody entirely knows the number that will come from Bulgaria and Romania."
He said he had had "no discussions with the Home Office with regards to the numbers".
But he said individual councils were taking action to prepare for a possible large increase in the numbers of people moving to the UK from the two countries next year.
"We do know of a number of boroughs that have a higher than average number of Romanians so I would expect to see influxes in the east of London, which is predominantly where they are now," he said.
Social housing problems
The last Labour government vastly underestimated the numbers of people from the first wave of eastern EU member states who would move to the UK, nearly a decade ago.
And Mr Pickles said he would not quote any figure until he was sure about it: "I have seen figures. I wasn't confident of those figures. I have asked for a further explanation and when I've got that explanation and when I feel confident about the figures I'll talk about them."
But, he added: "Given that we've got a housing shortage, any influx from Romania and Bulgaria is going to cause problems - it's going to cause problems, not just in terms of the housing market but also on social housing markets."
The Home Office has not produced an official estimate of how many of the 29 million Romanian and Bulgarian citizens will take advantage of their new freedoms when controls are lifted.
But some experts predict a large number of immigrants from Eastern Europe, which Labour has warned could put pressure on British jobs and wages.
The government's migration advisory committee says there is evidence that Bulgarians would move to Britain because of its stronger economy, and it was "plausible" Romanians would come for the same reason.
Home Secretary Theresa May said in November that temporary curbs could not continue under EU law and the government was looking instead at limiting access to benefits and the NHS to reduce the "pull factors" that encouraged migrants to come to the UK.