More studies are needed into the risks of mobile phones on sperm counts, warn researchers.
A review of the evidence, by the University of Exeter, suggested sperm number and movement were affected by keeping mobile phones in pockets.
However, one sperm scientist said the evidence was still too sketchy and his phone was staying in his pocket.
The study, published in the journal Environment International, suggested electromagnetic radiation was to blame.
It analysed 10 separate studies on sperm quality involving 1,492 men. These included laboratory tests on sperm exposed to mobile phone radiation and questionnaires of men at fertility clinics.
Lead researcher Dr Fiona Mathews told the BBC that all but one of the studies showed a link between mobile phone exposure and poorer sperm quality.
She added: "The studies are coming out with a consistent message that sperm motility declines with exposure to mobile telephones and similarly proportion which are alive, it's about an eight percentage point fall.
"I think for your average man there's certainly no need to panic, if you already know you have a potential fertility issue then it might be an additional thing to consider - just as you might change your diet - you might want to change where you keep your phone."
She acknowledged criticisms from other scientists about the quality of the evidence saying she "absolutely calls for more research".
Dr Mathews concluded: "This is interesting, but we're obviously not saying that everyone who carries a phone in their pocket is going to become infertile."
How sperm would be damaged by mobile phones is unclear.
Ideas include radio-frequency electromagnetic radiation from the phone disrupting the cycle of sperm production or damaging the DNA. Another suggestion is that heat either directly from the phone or through the radiation, may affect the sperm.
Dr Allan Pacey, from Sheffield University, who researches sperm, remains unconvinced, saying the quality of the evidence is poor and he would not change where he kept his phone.
He told the BBC: "There has been concern for some time about whether keeping a mobile phone in a trouser pocket might affect semen quality and male fertility in some way.
"There have been some crazy and alarming headlines, but, in my opinion, the studies undertaken to date have been somewhat limited in scope because they have either irradiated sperm kept in a dish or they have made assessments of men's phone habits without adequately controlling for confounding variables, such as other aspects of their lifestyle.
"What we need are some properly designed epidemiological studies where mobile phone use is considered alongside other other lifestyle habits.
"Until that time, I will be continuing to keep my iPhone in my right-hand trouser pocket!"