As many as a quarter of men have a genetic change which makes them less fertile than usual, research suggests.
The discovery could lead to a new screening test to identify those who will take longer to father a child, experts report in the journal Science Translational Medicine.
The change is in a gene that codes for a key protein found on the outside of sperm.
Sperm lacking in the substance find it harder to swim to the egg.
Researchers believe a man with the altered gene can still get his partner pregnant, but this will take longer than usual.
Dr Edward Hollox of the University of Leicester is a co-author of the study.
He told the BBC: "If you've got this gene variant you should allow that little bit longer if your partner's planning to get pregnant.
"It takes two - it's the genetic variation in a man that affects fertility in this particular case."
He said the discovery raised the possibility of a new test to identify couples who might need fertility treatment.
"It's another tool in the toolkit of fertility treatment," he said.
The genetic change is in a gene called DEFB126, which codes for a protein that clings to sperm, helping them swim through the woman's body to fertilise the egg.
Researchers believe men with the defective gene have sperm that find it harder to make their way through mucus, causing low fertility.
A study of more than 500 married couples in China found that women who had partners with two copies of the defective gene (one from the mother and one from the father) were less likely to get pregnant.
The women also took longer to get pregnant by a couple of months.
Further studies, carried out in people from the US, UK, China, Japan and Africa, found the gene mutation is common around the world.
About half of all men carry one copy of the defective gene; while a quarter have two defective copies.
Commenting on the study, Dr Allan Pacey, Senior Lecturer in Andrology at the University of Sheffield, said:
"We actually understand very little about the subtle molecular events which occur in sperm as they make their journey through the woman's body to fertilise an egg.
"We know even less about how a man's genes may contribute to how his sperm work, in the absence of an obvious defect that we can see down the microscope.
"Therefore, this paper is an important step forward and makes a significant contribution to our sperm-knowledge."