Communities Secretary Sajid Javid has attacked "nimbyism" as he called for a million new homes to be built by 2020.
While everyone agreed on the need for new homes, "too many" opposed them in their own community, Mr Javid told the Conservative Party conference.
While there were valid reasons for stopping developments, on grounds of size and taste, he said the UK couldn't afford to build so few homes.
He announced the "largest state-backed" housing programme since the 1970s.
Addressing activists in Birmingham, Mr Javid said there had been "massive progress" since 2010 but there was still a "long, long way to go" and the simple fact was that too many people were priced out of housing market and had little chance of owning their home.
The 170,000 homes built in England last year, he said, was simply not enough to keep up with demand.
Pledging to tackle the obstacles to new schemes, he called on developers to "release their stranglehold" over land and stop building up so-called land banks.
And targeting 'nimbyism' (which stands for 'not in my backyard') he said local communities had to reflect on the UK's national priorities not just their own.
"Everyone agrees we need more homes, but too many object to them being next to us," he said.
"We have got to change that attitude. Of course there are sometimes some valid reasons for opposing some local planning applications - if they are in the wrong place, if there is not enough infrastructure or if they are just plain ugly, but all of us we have a duty to think of the long-term consequences of every decision we make.
"We have a responsibility to build more houses, a responsibility not just to our constituents but to the next generation."
Unveiling what he described as "unprecedented" plans to "open up the market", he said the government would borrow £2bn to support the "Accelerated Construction" scheme, which aims to get houses built on publicly-owned "brownfield" land available for swift development. Mr Javid said the cash would encourage new developers to build up to 15,000 homes by 2021.
There would also be a £3bn Home Building Fund to provide loans to stimulate projects, which he said would build more than 25,000 homes by 2020, with a long-term goal to build more than 200,000.
"This is just the beginning," he added. "We will publish a housing white paper later this year with further significant measures all helping us towards our ambitions of a million new homes by 2020."
The Local Government Association, which represents councils in England and Wales, said more attention must be given to tackling the "growing backlog" in developments which had received planning permission but which had yet to get under way.
"Councils are approving nine in 10 planning applications yet our recent analysis also shows there are hundreds of thousands of homes with planning permission which are still waiting to be built," said its chair Lord Porter. "Councils need more powers to encourage developers to build homes more quickly."
The Green Party said the plans failed to tackle the "underlying causes" of the housing crisis, such as the sale of council homes and "tax breaks" for buy-to-let investors, while the Liberal Democrats dismissed the announcements as a "drop in the ocean".
"If the government wants to solve our country's housing crisis they will have to do much, much more than the level of investment announced today," said its leader Tim Farron.
"Experts say we need to be building 300,000 homes a year- double what we currently build. There is a huge task ahead and this announcement is a drop in the ocean."