The brain's ability to function can start to deteriorate as early as 45, suggests a study in the British Medical Journal.
University College London researchers found a 3.6% decline in mental reasoning in women and men aged 45-49.
They assessed the memory, vocabulary and comprehension skills of 7,000 men and women aged 45 to 70 over 10 years.
The Alzheimer's Society said research was needed into how changes in the brain could help dementia diagnoses.
Previous research had suggested that cognitive decline does not begin much before the age of 60.
But the results of this study show that it could in fact begin in middle age.
This is important, the researchers say, because dementia treatments are more likely to work at the time when individuals start to experience mental impairment.
The UCL researchers tested the cognitive functions of 5,198 men and 2,192 women aged 45 to 70, who were all UK civil servants, from 1997 to 2007.
Individuals were tested for memory, vocabulary and aural and visual comprehension skills.
Differences in education level were taken into account.
The results of the tests show that cognitive scores declined in all categories except vocabulary - and there was a faster decline in older people.
The study found a 9.6% decline in mental reasoning in men aged 65-70 and a 7.4% decline for women of the same age.
For men and women aged 45-49, there was a 3.6% decline.
Professor Archana Singh-Manoux from the Centre for Research in Epidemiology and Population Health in France, who led the research team at University College London, said the evidence from the study showed that dementia involved cognitive decline over two to three decades.
"We now need to look at who experiences cognitive decline more than the average and how we stop the decline. Some level of prevention is definitely possible.
"Rates of dementia are going to soar and health behaviours like smoking and physical activity are linked to levels of cognitive function.
"It's important to identify the risk factors early. If the disease has started in an individual's 50s but we only start looking at risk in their 60s, then how do you start separating cause and effect?"
Dr Anne Corbett, research manager at the Alzheimer's Society, said the study added to the debate on when cognitive decline began, but it left some questions unanswered.
"The study does not tell us whether any of these people went on to develop dementia, nor how feasible it would be for GPs to detect these early changes.
"More research is now needed to help us fully understand how measurable changes in the brain can help us improve diagnosis of dementia."
Dr Simon Ridley, head of research at Alzheimer's Research UK, said he wanted to see similar studies carried out in a wider population sample.
He added: "Previous research suggests that our health in mid-life affects our risk of dementia as we age, and these findings give us all an extra reason to stick to our New Year's resolutions.
"Although we don't yet have a sure-fire way to prevent dementia, we do know that simple lifestyle changes - such as eating a healthy diet, not smoking, and keeping blood pressure and cholesterol in check - can all reduce the risk of dementia."
Professor Lindsey Davies, president of the Faculty of Public Health, said that people should not wait until their bodies and minds broke down before taking action.
"We need only look at the problems that childhood obesity rates will cause if they are not addressed to see how important it is that we take 'cradle to grave' approach to public health."