Proposed changes to planning policy would be "catastrophic" for rural areas, rural housing and planning authorities are warning.
The government wants to scrap the requirement for developers to provide affordable housing on smaller developments.
Campaign groups say that supplies of affordable housing would "dry up".
The government argues that the change would remove red tape and encourage more house building.
The warning on affordability comes from Action for Communities in Rural England (ACRE), the Rural Services Network, along with England's National Parks.
At the moment to get planning permission, local authorities in England can compel developers to build a certain amount of affordable housing.
The proposal is to remove this obligation on sites of fewer than 10 units.
Government figures show that around two thirds of affordable houses in rural areas last year were built on small sites.
Nick Chase from ACRE said: "This change would have a catastrophic effect on the numbers of affordable houses coming forward for local communities.
"It also flies in the face of allowing local communities to take responsibility for the numbers and types of houses that they want." The organisation wants villages with a population of less than 3000 to be exempt."
For the first time the heads of all ten National Parks in England have come together to personally sign a letter to the government. It says the measure "puts at risk" their affordable housing supply.
Ghost towns and villages
They believe that in rural areas, the open market is already failing to provide enough homes that people on average salaries can buy or rent.
According to Chief Executive of the Yorkshire Dales National Park, leaving housing purely to the open market will lead to "ghost towns and villages".
David Butterworth told the BBC, "Between 2001 and 2011 in the Yorkshire Dales there were 1000 homes built or converted. However the population increased by only 100. That's an extraordinary statistic. The proportion of housing that was second home or holiday cottage rose from 15% to 23%. Is this the kind of society that we want?
The issue underlying all this is the viability of rural areas in general. Of getting a good demographic mix which is vital for communities to grow."
According to the National Housing Federation, on average house prices in rural areas are 11 times the average salary. Around half (44%) of the 50 most unaffordable places to live in England outside of London are in rural areas.
Direct government grants for new housing have declined significantly. In addition, councils build far fewer homes than they used to. In their place is 'affordable housing'. This is a generic term that includes shared ownership, social rented - where rental rates are kept low - and "Affordable Rent", where tenants can be charged up to 80% of market rate.
Most of this new affordable housing comes from planning agreements between councils and private developers under a "section 106 agreement". On developments of fewer than 10 homes, they would no longer have any say.
Gareth Wooler is a property developer based in Keighley. His company employs 200 people. He told the BBC that it was "really very difficult to deliver affordable homes quickly and efficiently on sites of fewer than 10 homes. It doesn't deliver economies of scale."
He said that many people didn't want to buy or rent a house next to a social tenant. To get permission to build eight homes on a site, his company recently had to make one an affordable home. "There's an issue of saleability. Sales on either side of the social house were at a reduced price because of the stigma attached," he explained.
"Landowners are sitting on land. If we could build up to ten units without any affordable housing, then landowners on those sites would be much more keen to sell."
The policy change would apply to those wishing to build their own homes and would not apply to "rural exception sites".
According to Communities Secretary Eric Pickles MP, these changes would also apply to those wishing to build their own homes. He said: "There are too many levies and charges on housing. By cutting these, we can help support hard working families, give builders a boost and build more homes"
However rural housing organisations also fear the consequences. Teresa Snaith from Home Group explained: "The need for rural housing is great. Without the kind of housing we provide, places would become second home or commuter villages."