Scientists say that culling a small number of badgers risks increasing the spread of TB infection to cattle.
The research suggests that some farmers who have allegedly killed badgers on their property could be making things worse for themselves and neighbours.
The findings also indicate that Northern Ireland's badger culling strategy could be counter-productive.
The study has been published in Proceedings of the National Academy of Sciences (PNAS).
The deputy president of the National Farmers Union (NFU) said that all options should be considered to eradicate cattle TB.
Prof Rosie Woodroffe, who led the research, said that Northern Ireland's Department for Agricultural Development (Dard) should consider dropping its plans for selective culling of badgers.
"It is more likely to make things worse," she told BBC News. "My advice would be to trial vaccination."
Prof Woodroffe's analysis suggests that even culling a small number of badgers increases the movement of badgers - an effect known as perturbation - which increases rather than decreases the risk of cattle becoming infected.
Dard said that it would continue with its pilot cull, known as the Trap, Vaccinate and Remove (TVR) Research Project, because it believed that the data used in the new study was gathered from culls in England between 1986 and 1998 and could not be translated to the different terrain of Northern Ireland.
"This is why it is essential for the the TVR Research Project to proceed to see if perturbation is a factor here following badger removal," the department said in a statement to BBC News.
The results also suggest that farmers who are allegedly culling badgers may be doing more harm than good, according to Prof Woodroffe.
"We are aware that the illegal culling of badgers is widespread in England. What this work shows is that it is causing perturbation and so is probably causing an increase in badger TB."
Dard plans to cull badgers in a 100 sq km area in Bainbridge in County Down. The strategy involves first trapping badgers, testing them for infection, killing those that test positive and vaccinating those that seem to be infection-free.
It is a more targeted approach than used by the Department for Environment, Food and Rural Affairs (Defra) in two pilot sites in Somerset and Gloucestershire which involved attempting to kill up to 70% of badgers in the areas without testing to see if they were infected.
It was hoped that the more selective approach adopted in Northern Ireland would be more effective.
However, the new study suggests that even if one badger is killed in a sett then there is an increased risk of TB infection spreading to cattle, rather than it declining.
The uncertainty is compounded by the facts that the test can detect only half the number of infected badgers and vaccination is not 100% effective, according to Jon Bielby who also worked on the study.
"The policy appeals to a lot of people as it avoids indiscriminate culling. But you are not going to catch all the infected badgers and you are not going to detect all the infected badgers. Combined with the fact that it increases movement [this] makes it a very risky strategy," Mr Bielby told BBC News.
The deputy president of the National Farmers Union, Minette Batters, said that all options should be considered to eradicate TB in cattle, including the large scale culls adopted in England.
"The Chief Vet has said that badger culling over a four-year period in the pilot areas in Somerset and Gloucestershire will help to reduce TB in cattle.
"We remain confident that these pilots will help to deliver a reduction of TB in cattle and it is vital that they are allowed to be successfully completed so they can deliver the biggest benefit and help us eradicate this terrible disease."
Dard says it has always been aware of the potential for badger movement.
"We are introducing certain measures to monitor and mitigate against the effects of perturbation should it happen here. This year, all captured badgers under TVR will be vaccinated and released. This is in preparation for badger removal from next year and is intended to build up some immunity to bovine TB in the badger population in advance of badger removal."
The Northern Ireland Badger Group has backed Dard's approach and said in a statement that the new research had not altered its view.
"One of the concerns others have raised about this project is the potential for perturbation. It is our understanding that no badgers will be removed in the first year of the project to provide baseline data and to mitigate any anomalies that badger removal might cause," it said.
But Mark Jones, the executive director of the Humane Society International UK, said that the new research confirmed his concerns about the impact that culling has on badgers.
"Farmers who decide to target badgers on their own land are not only breaking the law, but also increasing the risk of infection in their cattle, and those of their neighbours," he explained.
"It's high time Defra and the National Farmers Union started to listen to the scientists rather than their own flawed rhetoric, and abandoned badger culling in favour of the kinds of cattle-focused measures that are working so well in Wales, and the development and deployment of vaccines for both badgers and cattle."
Follow Pallab on Twitter