Exercise programmes for people with mild to moderate dementia "don't work", according to researchers writing in the British Medical Journal.
They found no improvements in thinking skills or behaviour in more than 300 people in their 70s who did aerobic and strength exercises over four months.
On the plus side, their physical fitness did improve, the study said.
The Oxford researchers said future trials should explore other forms of exercise.
Gentle, regular exercise was a good thing, they added, and there was no reason for anyone to stop exercising.
But at present, structured NHS exercise programmes for dementia patients did not appear to be a good investment.
In the study, 329 dementia patients took part in gym sessions lasting 60-90 minutes twice a week for four months.
They spent at least 20 minutes on a fixed cycle and lifted weights while getting out of a chair.
They were encouraged to do exercises at home for another hour each week.
The exercise group was then assessed and compared with a group of 165 people with dementia who received their usual care.
Prof Sallie Lamb, lead study author and professor of rehabilitation at Oxford University, said the results showed that people who had had dementia for two or three years could follow simple exercise instructions and improve their fitness and muscle strength.
"But these benefits do not, however, translate into improvements in cognitive impairment, activities in daily living, behaviour, or health-related quality of life," she said.
After 12 months, researchers found that cognitive impairment had declined in both groups, with the exercise group slightly worse off - but the difference was small.
Prof Martin Rossor, professor of clinical neurology at University College London, said the results weren't surprising given degeneration of brain cells started many years before symptoms began in Alzheimer's disease, for example.
"So, the message remains that exercise is good, but to start an exercise regime once the disease is well established may be of limited value," he added.
Enjoyment of exercise
Dr Sara Imarisio, head of research at Alzheimer's Research UK, said other forms of exercise could have different effects and this should be explored in future research.
She said there were many benefits to physical exercise, apart from simply health ones.
"For many people, exercise can be a source of enjoyment and provide valuable opportunities for social interaction," she said.
"These considerations can apply to people living with dementia just as much as they do to anyone else."
Exercise is still thought to be one of the best ways to reduce the risk of getting dementia in healthy older adults.
But this research suggests larger trials are needed to work out an effective exercise programme for brain health in those who already have the condition.