A woman with a neuro-muscular condition has described being chased in a car park and told she was abusing the blue badge disabled parking scheme.
Sara Flay, from Neath, needs support from ankle and leg braces but her disability is not immediately obvious.
"The incidences were so negative at one point, that I stopped reapplying for my blue badge," she said.
Welsh ministers and councils said they were working with disabled people and groups to address the issue.
Blue badges are parking permits allowing drivers or passengers with mobility issues to park nearer to where they are going - usually in specially marked disabled parking bays.
The criteria for getting a blue badge includes receiving disability benefit, having severe sight impairment or being assessed as having seriously limited mobility.
The scheme has recently been extended to include people unable to plan or follow a journey without help from someone else, such as those with learning disabilities or dementia.
Ms Flay has needed support from ankle braces for most of her life and recently leg braces.
She told BBC Sunday Politics Wales her disability was not necessarily obvious, and she gets stares from people on a regular basis when using her badge.
"I've had one incidence of being chased across a car park and shouted at and told I was abusing the system, because I wasn't actually disabled, even though I had a blue badge on my car," she said.
"That made me really anxious and stressed.
"For me, the incidences were so negative at one point, that I actually stopped reapplying for my blue badge, and I didn't have a blue badge, even though I was still disabled and my condition is progressive, but it meant that I was just struggling, because I'd rather struggle and deal with it my myself everyday than having people being abusive or commenting towards me."
Ms Flay has since started using her badge again.
Disability Wales chief executive Rhian Davies said that despite the eligibility being widened, some people assume you only get a blue badge if you are unable to walk.
She said eligible blue badge holders were being questioned or challenged "in a way that is quite aggressive, using terms of abuse, questioning people 'why have you got a blue badge?'
"We are hearing about disputes in places like supermarket car parks," she said.
"This issue needs to be taken more seriously than it is, because many disabled people are telling us that they are now afraid to either use their blue badge or even go out and about because they are afraid of the abuse and the harassment that they are going to experience."
The Welsh Government is responsible for the Blue Badge scheme in Wales, and local councils administer it.
Disability Wales said a campaign by the Welsh Government and councils was needed to explain the barriers blue badge holders face on a day-to-day basis.
A Welsh Government spokesperson said: "We will continue to work with disabled people, and groups who represent the interests of disabled people, on these issues through our new Framework 'Action on Disability: the Right to Independent Living' and through our Disability Equality Forum and Hate Crime Criminal Justice Board.
"Anyone who finds themselves subjected to such attacks should be aware that abuse of this manner might be a criminal offence and should therefore be reported to the police."
A Welsh Local Government Association spokesperson said: "In applying for a blue badge, all holders will have met agreed criteria, ensuring that the scheme is wholly reserved for those who experience limited mobility and who qualify on an automatic, discretionary or temporary basis."