While mums-to-be and women trying for a baby should limit their caffeine intake, a couple of cups of tea or coffee a day is fine, say experts.
Their comments come as a new research paper in a medical journal suggests there is no safe level in pregnancy.
But the experts say that is alarmist.
The NHS and many other organisations say consuming 200mg or less a day should not pose any significant risk in terms of miscarriage or growth of the baby while in the womb.
The stillbirth charity Tommy's has a caffeine intake calculator to help women keep track of their consumption.
The controversial research paper, published in BMJ Evidence Based Medicine, looked at 48 studies on the topic.
The author of that paper, Prof Jack James, a psychologist at Reykjavik University in Iceland, acknowledges that the work is observational, so can't prove definitively that any caffeine in pregnancy is harmful.
But he says his analysis, which links caffeine with harm, suggests avoiding drinks like tea and coffee entirely would be the best advice for mums-to-be and women trying to get pregnant.
Other experts strongly disagree, saying this is overkill.
Just as the NHS does, the European Food Safety Authority and the American and UK Colleges of Obstetricians and Gynaecologists recommend limiting, but not eliminating, caffeine consumption during pregnancy.
Dr Luke Grzeskowiak, a pharmacist at the University of Adelaide, Australia, said the research paper was "overly alarmist" and inconsistent with accepted evidence.
"There are so many dos and don'ts associated with pregnancy and the last thing we need is to cause unnecessary anxiety. At the end of the day, women should be reassured that caffeine can be consumed in moderation during pregnancy."
Prof Andrew Shennan, professor of obstetrics at Kings College London, said some of the studies in the analysis may be flawed because they rely on women recalling caffeine intake. Also, he said, it is difficult to exclude other risk factors that tea or coffee drinkers might be indulging in, such as cigarette smoking.
He said: "Caffeine has been in human diets for a long time.
"Like many substances found in a normal diet, harms in pregnancy can be found with high doses.
"However the observational nature of this data with its inherent bias does not indicate with any certainty that low doses of caffeine are harmful, and the current advice to avoid high doses of caffeine are unlikely to change."