Unhealthy habits which increase heart disease risk could also be affecting brain function in people as young as 35, a study suggests.
Dutch researchers examined almost 3,800 people aged 35 to 82, testing memory, planning and reasoning.
High cholesterol and heavy smoking were linked to poorer performance in the tests across all ages.
The Alzheimer's Society said it showed what was bad for the heart was also "bad for the head".
Writing in the journal Stroke, the researchers from the University Medical Centre in Groningen said there was already evidence that decline in brain function could be seen by the age of 45.
To check if signs could be detected even earlier, the team assessed participants' mental abilities - cognitive function - using a comprehensive test.
They also looked at risk factors for heart problems - including smoking, high cholesterol, high blood pressure, diabetes and being overweight.
This enabled them to compile a "risk profile" for each participant.
Those with the most serious risk for heart problems fared around 50% worse on the cognitive tests than those with the lowest-risk profile - across the age range.
And those who smoked one to 15 cigarettes per day had a cognitive score that was on average two-point-four points lower than non-smokers, while those who smoked more than 16 cigarettes a day had on average a three-point-four point lower score.
Dr Hanneke Joosten, who led the study, said: "Young adults may think the consequences of smoking or being overweight are years down the road, but they aren't.
"Most people know the negative effects of heart risk factors such as heart attack, stroke and renal impairment, but they do not realise it affects cognitive health.
"What's bad for the heart is also bad for the brain."
Doug Brown, director of research at the Alzheimer's Society, said: "We all know that smoking, high blood pressure and high cholesterol levels are bad for our heart.
"This study adds to the wealth of evidence that also suggests they can be bad for our head too.
"As the study shows it is important that people think about their lifestyle habits and their effects a lot earlier than they may have thought they needed to.
"Telling ourselves we'll quit smoking tomorrow, or eat healthily next week could be causing more damage than people realise."