One in four patients with secondary breast cancer had to visit their GP three or more times before they got a diagnosis, a survey suggests.
A breast cancer charity said there should be more awareness that the disease can spread to other parts of the body.
GPs said they were doing their best for patients but symptoms could be difficult to spot.
In the UK, 35,000 people are living with the incurable form of the disease.
Breast Cancer Now said it was "unacceptable" that some people whose cancer had spread were not getting early access to treatments which could alleviate symptoms and improve their quality of life.
"For too long now, the worrying perception that everyone survives breast cancer has masked the heartbreaking reality for 11,500 families in the UK that lose someone they love each year," the charity said.
The advanced, or metastatic form of the disease means the cancer has spread through the blood and created secondary tumours in the bones, liver, lung or brain.
It cannot be cured and patients stay in treatment for the rest of their lives.
'I felt stupid and naive'
Jo Myatt, 43, from Chorley, went to see her GP five times over a number of years before discovering that breast cancer had spread to her liver and bones.
"I felt like a hypochondriac for going all the time," she says.
Her symptoms started with missed periods and nausea before becoming more serious and persistent, leaving her unable to move her neck.
But no-one had mentioned secondary breast cancer as a possibility when she was diagnosed 10 years previously.
"I was devastated. I felt stupid and naive," Jo says.
"People don't realise it can spread to other areas of the body and you can have nothing in your breast."
Jo is now on her fourth treatment and waiting anxiously for the results of her latest scans.
She knows she is eventually going to run out of options, but she urges other women to speak up.
"If you're worried about pain, make doctors aware of your history.
"You're not going mad - the sooner you find the secondary, the better health you're going to be in and the more it can be contained."
What are the symptoms of secondary breast cancer?
They can vary depending on where the cancer has spread to, but common signs and symptoms include:
- unexpected weight loss or loss of appetite
- discomfort or swelling under the ribs or across the upper abdomen
- severe or continuing headaches
- altered vision or speech
- feeling sick most of the time
- breathlessness or a dry cough
- loss of balance or weakness or numbness of the limbs
- any lumps or swellings under the arm, breastbone or collarbone
- pain in the bones (eg back, hips or ribs) that doesn't get better with pain relief and may be worse at night
Cancer Research UK lists more symptoms of secondary breast cancer on its website.
Breast Cancer Now's survey of 2,100 people in the UK with secondary breast cancer found that just 13% were told of the symptoms to look out for if their cancer spread.
And four in 10 said they felt their symptoms had not been taken seriously before they were diagnosed.
Breast cancer is the most common type of cancer in the UK, with around 55,000 new cases every year.
Although survival rates have significantly improved over the last 40 years in the UK, there are still around 11,500 deaths from breast cancer each year - mostly from secondary breast cancer.
'Working flat out'
Prof Helen Stokes-Lampard, from the Royal College of GPs, said: "GPs and our teams are working flat out to do the very best we can for all our patients.
"We understand the importance of timely cancer diagnosis and are highly trained to identify possible symptoms of cancer and its recurrence."
But she said some symptoms were "very difficult to interpret because they are vague in the initial stages" or similar to other, more common conditions.
Prof Stokes-Lampard called for GPs to have better access to the right diagnostic tools and training to use them.