Private clinics have a "moral duty" to remove banned PIP breast implants from women they operated on, the government says.
The NHS will cover the costs for women who had the implants fitted by the health service and who are anxious to have them removed, it added.
The NHS will also remove the implants if the private clinic no longer exists or refuses the patient.
Around 40,000 women in the UK have been fitted with them.
It is thought 95% of women had the operation privately, 5% on the NHS.
The French authorities have offered to pay for implants to be removed due to a high risk of them rupturing.
Czech and German health authorities both recommended on Friday that women in those countries with PIP implants should have them removed.
The UK review was ordered because of conflicting data about the risk of the implants rupturing and leaking non-medical grade silicone into the body.
The French authorities quoted a rate of 5%. The UK's Medicines and Healthcare products Regulatory Agency (MHRA) initially said the rate was 1% - in line with other implants.
There were reports of rupture rates of 7% from one cosmetic surgery group, Transform. However, it says that rate was based on just seven out of 108 patients it fitted with PIP implants since 2005.
Transform chief executive Nigel Robertson accused the MHRA of a failure in its duty to "monitor and routinely audit" the use of the implants.
"The NHS and the cosmetic surgery industry used these products because they had been approved by the MHRA for medical use and carried a CE mark to that effect," he said.
The agency must bear responsibility for the current situation, he added.
But in response, the MHRA said the CE mark had to be authorised though an independent third-party organisation in all but the lowest risk products.
It added: "The MHRA's key role is to monitor and investigate reports of device-related adverse incidents and take appropriate action to prevent their recurrence."
The agency said it had continually monitored the safety of the PIP breast implants and immediately advised clinicians not to implant them once the French authorities informed them of the use of unauthorised silicone gel.
It also said it commissioned toxicity tests on the gel, the results of which concluded there was "no safety issue" related to the material.
The health secretary, Andrew Lansley, said: "The data available to the experts has not been good enough to enable them to give a clear recommendation of the risk posed by PIP implants."
The review committee could not determine if the rupture rate was higher for PIP implants.
However, it said it was "undeniably the case" that the implants were not medical grade and should "not have been implanted in women in the first place.
"We have always recommended that women who are concerned should speak to their surgeon or GP. The NHS will support removal of PIP implants if, after this consultation, the patient still has concerns and with her doctor she decides that it is right to do so."
He added: "We believe that private healthcare providers have a moral duty to offer the same service to their patients that we will offer to NHS patients - free information, consultations, scans and removal if necessary.
The implants of NHS patients will be replaced. Private patients refused by their clinics will be able to have their implants removed by NHS surgeons, but will not be offered replacement cosmetic implants.
The government said it will: "Pursue private clinics with all means at its disposal to avoid the taxpayer picking up the bill".
Prof Sir Bruce Keogh, the NHS medical director who led the review, said: "On the basis of the information we have, we do not think it is necessary to recommend the routine removal of these implants.
"But we understand that some women will be very concerned so we support the government's position that the NHS will support removal of PIP implants if the patient has concerns and with her doctor she decides that it is right to do so."
Prof Keogh told BBC Newsnight the NHS "is offering something that is a much better solution... of informed, personalised decision making".
A joint response by The Association of Breast Surgery, British Association of Plastic and Reconstructive Surgeons, British Association of Aesthetic Plastic Surgeons, Federation of Surgical Speciality Associations and the Royal College of Surgeons said the announcement was welcome.
"The news that all women who have received a PIP breast implant will be provided with support, and that all NHS patients will be offered monitoring, advice and remedial treatment if desired is welcome news.
"Surgeons' organisations applaud those private cosmetic clinics who have committed to offer treatment to their patients free of charge and fully back the government call for the rest to follow suit.
"This situation raises again the need for better regulation and surveillance for all surgical implants and the surgical profession believes mandatory databases should be the next step - not least because this issue has exposed poor record keeping."
Sally Taber, the director of the Independent Healthcare Advisory Services, which represents the majority of private clinics, said 60% of the implants were fitted by four companies: Harley Medical Group, The Hospital Group, Transform and Linia Cosmetic Surgery.
She said the companies were considering their position, wanted to examine how the measures would be paid for and were seeking further discussions with the Department of Health.
Spire Healthcare says it will follow the governments recommendations. Its clinical director, Dr Jean-Jacques de Gorter, said: "We have a duty of care to our patients.
"We believe all private providers should step up and deliver on their duty of care to patients. Every woman in the UK who has PIP implants should be given the right to discuss having them removed if they wish."
Nuffield Health's group medical director Dr Andy Jones said: "Any patient who had a PIP implant in one of our hospitals can be assured that they will continue to receive the help and attention they need.
"We believe that there is a strong case for the private healthcare industry to pull together to resolve this matter in the interests of all patients."