All patients getting cancer treatment should be told to do two and a half hours of physical exercise every week, says a report by Macmillan Cancer Support.
Being advised to rest and take it easy after treatment is an outdated view, the charity says.
Research shows that exercise can reduce the risk of dying from cancer and minimise the side effects of treatment.
The Department of Health says local initiatives can get people moving.
Macmillan's report, Move More, says that of the two million cancer survivors in the UK, around 1.6 million are not physically active enough.
Adult cancer patients and cancer survivors should undertake 150 minutes of moderate intensity physical activity per week, the reports says, which is what the Department of Health guidelines recommend.
In the report, the American College of Sports Medicine also recommends that exercise is safe during and after most types of cancer treatment and says survivors should avoid inactivity.
Getting active, the report says, can help people overcome the effects of cancer and its treatments, such as fatigue and weight gain.
"The evidence review shows that physical exercise does not increase fatigue during treatment, and can in fact boost energy after treatment."
"It can also lower their chances of getting heart disease and osteoporosis.
"Also, doing recommended levels of physical activity may reduce the chance of dying from the disease. It may also help reduce the risk of the cancer coming back."
Previous research shows that exercising to the recommended levels can reduce the risk of breast cancer recurring by 40%. For prostate cancer the risk of dying from the disease is reduced by up to 30%.
Bowel cancer patients' risk of dying from the disease can be cut by around 50% by doing around six hours of moderate physical activity a week.
Ciaran Devane, chief executive of Macmillan Cancer Support, said physical activity was very important to the survival and recovery process.
"Cancer patients would be shocked if they knew just how much of a benefit physical activity could have on their recovery and long term health, in some cases reducing their chances of having to go through the gruelling ordeal of treatment all over again.
"It doesn't need to be anything too strenuous, doing the gardening, going for a brisk walk or a swim, all count."
Traditionally cancer patients were told to rest after their cancer treatment, but the report says this approach could put cancer patients at risk.
Jane Maher, chief medical officer of Macmillan Cancer Support and a leading clinical oncologist said: "The advice that I would have previously given to one of my patients would have been to 'take it easy'.
"This has now changed significantly because of the recognition that if physical exercise were a drug, it would be hitting the headlines."
Martin Ledwick, head information nurse at Cancer Research UK, was a little more cautious.
"Anything that improves wellbeing and reduces treatment side effects for cancer survivors has to be a good thing.
"But the evidence that exercise has a bearing on survival is not conclusive. It is important to remember that no two cancer patients are the same, so rehabilitation programmes that include physical activity will need to be tailored to the individual."
A spokesperson from the Department of Health said it was vital that people with cancer are given the support to lead an active life.
"Physical activity and a healthy lifestyle can impact very positively on cancer outcomes and, as part of the National Cancer Survivorship Initiative, we are working with Macmillan to integrate physical activity services into cancer care pilot sites.
"Locally led initiatives such as Let's Get Moving are also well placed to signpost cancer patients to community-based physical activity opportunities."