Patients could be getting sicker and need more expensive treatment due to GP appointment difficulties, it has been claimed.
The Royal College of General Practitioners (RCGP) Wales said surgeries were "buckling under the strain of rising workloads".
It estimates 650,000 people in Wales found it difficult to get an appointment last year.
The Welsh government said the number of GPs and spending on them has increased.
But the RCGP says family doctors are struggling to cope with falling resources.
By 2017, it says up to 800,000 people in Wales could find it difficult to secure a convenient appointment.
RCGP Wales chair Dr Paul Myres, who represents 1,800 members from more than 2,000 GPs in Wales, said the worry for doctors was that some patients might not bother trying to secure an appointment.
He said: "Sometimes that's fine because many problems get solved of their own accord without anyone doing anything about it. Many conditions are self-limiting.
"But [it is] always a concern to us that people have difficulty getting access to us - that they delay their presentation until too late in the day and their condition then takes longer to treat.
"Just slightly over a third of people admitted they've have difficulty accessing their GP, so we already know that the service is not delivering as it should.
"We need to be available to see our patients roughly when they need us and that is not happening at the moment."
Dr Myres said retention and recruitment of GPs was a major issue in north Wales, while across Wales 23% of GPs were over 55 years old.
In the Cwm Taf health board area in the south east Wales valleys it was as high as 31%, he said.
He added: "When these doctors retire we have no-one to replace them. Who is going to care for our population?"
A Welsh government has described the RCGP's figures, extrapolated from year-on-year surveys, as "flawed".
It stressed that investment in general practice had increased by almost £150m over the last decade - to £469m.
A spokesperson said GP numbers were up over the same period by 11%.
"Where there are recruitment issues that need to be resolved, health boards are working with the profession to develop innovative ways of providing primary care services in future," said the government official.
"Making it easier for working people to get an appointment at a time convenient to them is something we set out as a key commitment of this government. The most recent figures show we are continuing to deliver on this commitment."