Intensive, high-yielding agriculture may be the best way to meet growing demand for food while conserving biodiversity, say researchers.
But their study says the approach makes sense only if it is linked to more wilderness being spared the plough.
Intensive farming is said to create high levels of pollution and damage the environment more than organic farming.
However, this report suggests that contrary to perceptions, this is not necessarily the case.
Organic groups though have rejected the report's findings.
Around the world sales of organic produce have boomed over the past 20 years as consumers have bought into the idea that the approach is good not just for their health but for the good of the planet, as well.
However, this study takes issue with that view.
The researchers measured the environmental costs - including what they term "externalities", such as greenhouse gas emissions, fertiliser and water use - of producing a given amount of food on both high-yield and low-yield farms.
Working with scientists in 17 organisations from around the globe, they analysed information from hundreds of investigations into four large food areas - Asian paddy rice, European wheat, Latin American beef and European dairy.
While the data available is limited, the researchers concluded that many high-yield systems are less damaging to the environment and use less land.
"If we are serious about keeping most of the species with which we share the planet, we are going to have to make the best use of the farmland we've already got rather than continue to expand," said lead author Prof Andrew Balmford, from the University of Cambridge.
"And that means we are going to have to get smart about high-yield agriculture."
The authors say the world needs to reduce consumption of beef and other high-impact foodstuffs if we are to have a sustainable future - but they believe their study shows that producing these materials with less land may be the best way of making space for nature and may also have fewer environmental costs than lower-yielding systems.
"We are certainly not pro conventional agriculture business as usual," said Prof Balmford. "'We're just pro looking at the numbers, and the numbers tell us that most biodiversity can't survive on any sort of farm."
When it comes to organic dairy farming in Europe, the researchers found that for the production of the same volume of milk, organic systems took up twice as much land and caused at least one-third more soil loss than conventional dairy farming.
"Across all dairy systems we find that higher milk yield per unit of land generally leads to greater biological and economic efficiency of production," said co-author Prof Phil Garnsworthy, from the University of Nottingham,
"Dairy farmers should welcome the news that more efficient systems have lower environmental impact."
One key element of this report is the link between intensive farming and leaving more land for wilderness.
The authors say there are various mechanisms that can be put in place to encourage farmers to take land out of production and reward them for the "public good" element that their lands provide, such as stemming the flow of floodwater, or absorbing carbon dioxide.
Environment Secretary Michael Gove has made this idea a key element of his proposed reforms for agriculture funding in the UK in the wake of Brexit.
Prof Balmford supports the approach.
"Land owners do produce very significant public goods that the market doesn't reward them for. Using some of the subsidy to do that offers considerable promise."
Organic group though are not impressed with the study's conclusions. They argue that the world produces enough food already and that the issues with feeding populations are economic and political, not agricultural.
"When the environmental and other damages caused by high energy and chemical inputs in non-organic are factored in - some of which the report's authors did not take into account - the research we've seen to date shows organic farming is more beneficial to society," said a spokesman for the UK's Soil Association.
"Organic methods show that it is possible to have long-term sustainable food production and environmental conservation; supporting 50% higher abundance of wildlife than non-organic and higher standards of animal welfare, while improving soil health and supporting soils to capture up to 450kg more carbon per hectare."
The study has been published in the journal Nature Sustainability.