Services for neurological conditions are poorly organised and do not meet patients' needs, says a report.
The Royal College of Physicians and the Association of British Neurologists say many patients with conditions like epilepsy or Parkinson's disease are unable to access specialist care.
They point to a lack of expert doctors in local hospitals and emergency departments.
The government agrees a shake-up of services is needed.
Neurological disorders are very common, making up about a 10% of GP consultations and emergency medical admissions.
The disorders include many different conditions, some very common, such as migraine and multiple sclerosis, and some rare, like motor neurone disease.
Together these conditions result in disability in one in 50 people in the UK.
But neurology services in the UK have mainly developed around large regional neurosciences centres with an emphasis on research and academic excellence.
This has left local services undertrained and understaffed, according to the report.
The UK also has fewer neurologists per head of population compared to other countruies - one per 125,000 in the UK compared to one per 40,000 in the US and the rest of Europe.
'Poor organised services'
Patients admitted to hospital with an acute neurological illness are rarely seen by a specialist neurologist.
In contrast, those admitted for a stroke and other acute medical emergencies usually see the right specialist without delay.
The report calls for an expansion and improvement of local services with a shift in emphasis from scheduled to emergency care.
The chairman of the working party who produced the report, Dr David Bateman, said: "The recommendations when implemented will substantially improve local services: many can be achieved at little extra cost mainly by reorganisation of services."
Steve Ford, Chair of the Neurological Alliance, said: "Patients with neurological conditions need to see the right specialist at the right time in the right place, but evidence shows clearly that this isn't happening due to poorly organised services and not enough specialist care.
"We welcome the report's timely publication and call on the government, at this important time of NHS reform, to place neurology at the top of its agenda."
A Department of Health spokesman said: "This is exactly why we need to modernise the NHS. Support for people with long-term neurological conditions has not been good enough.
"Our plans put patients firmly in the driving seat with more control over their care, and give clinicians the freedom to prescribe the treatment they feel most appropriate.
"Improving commissioning and more integrated services are key to ensuring better care for patients."