New research suggests that extreme weather events will keep people poor in many parts of the world.
The authors argue that where disasters like drought are prevalent, they can be the most important cause of poverty.
They say that up to 325 million people will be living in countries highly exposed to natural hazards by 2030.
If aid is not used to reduce these risks, the progress made in fighting poverty could disappear.
The report has been compiled by the Overseas Development Institute.
It examines the relationship between disasters and poverty over the next 20 years, using population projections, climate models and estimations of how governments can cope with extreme events.
The report suggests that up to a third of a billion people could be living in the 49 countries most exposed to the full range of natural hazards and climate extremes in 2030.
In sub-Saharan Africa 118 million people in poverty will face extreme events.
Drought means poverty
The big weather issues that will face most poor people are drought, extreme rainfall and flooding.
An analysis of the data from rural Ethiopia and Andhra Pradesh in India suggests that where there is a strong risk of drought, then drought is also the single most important factor in keeping people poor, outstripping ill health or dowry payments.
"We've often heard that ill health is the biggest cause for impoverishment," said Dr Tom Mitchell, the ODI's head of climate change.
"But in the data, in drought prone areas, the biggest cause is the drought - in areas exposed to these hazards, they are the key causes of impoverishment."
Developed countries haven't recognised the role that these extreme weather events have in keeping people poor, he says.
The big problem is that, at present, money tends to flow in response to disasters, not to prevent them.
Dr Mitchell says the recent Cyclone Phailin in India is a good example.
"The very fact that it killed so few people means that the chances of raising big finance for recovery efforts are going to be pretty slim. It has not got the big numbers attached to it," he said.
"I think there's a direct link between the ability to raise finance and the number of people killed. It's a perverse incentive."
Part of the problem is that donor countries are not prioritising aid at the countries that need it most, in terms of disaster risk reduction.
"We've tended to provide much more financial support to a set of middle income countries, who can manage it better like the Philippines, Mexico and Indonesia who made really great strides in protecting their populations," said Dr Mitchell.
"What we've not done is focus on the poorest countries, the ones most exposed to issues like drought, for example, sub Saharan Africa, we've almost missed it off."
The authors of the report argue that the way that vulnerable countries spend their money needs reforming too. Too often the money is spent on the capital city or on infrastructure and not on the poorest people.
The report calls for the post-2015 development goals to include targets on disasters and climate change, to recognise the threat they pose to eradicating poverty by 2030.
"If the international community are serious about ending extreme poverty they need to get serious about reducing disaster risk for the poorest people," said Dr Mitchell.
"At the moment that's not happening, so the chance of ending extreme poverty is pie in the sky, it is just not going to work."
Follow Matt on Twitter.