There have been big advancements in the acceptance of the transgender community in recent years but some divisions over their rights remain.
"If I'd been an ordinary person and I had daughters, I'd think 'what's this? Men coming into the toilets? I don't like the sound of that'," says campaigner Paris Lees, who is transgender.
She supports trans people having increased access to female facilities - such as toilets or changing rooms - but understands some people have legitimate concerns.
She tells the Victoria Derbyshire programme this is because some people do not really understand what is happening or are confused, while other people are playing on ignorance and prejudices and "whipping up fear".
"It is that playing on people's prejudices and [in the same way] there was no evidence that gay people would harm children, or children adopted by same sex couples are any more likely to be in danger," Ms Lees says.
"I think it is really irresponsible whipping up this fear against us based on zero evidence."
Some of the women raising concerns say they are being silenced when they just want an open discussion.
Community organiser and campaigner Pilgrim Tucker says: "Let's slow down and really talk about why women might feel cautious about having trans women in their spaces, places where they will feel vulnerable."
The issue needs "lots of careful thought" she says, due to the differences between trans women and people who were born biologically female.
The way trans people are legally recognised is another issue, on which the government is currently consulting.
Currently, anyone who wants to legally change their gender in the UK has to apply for a Gender Recognition Certificate, but many feel the process is long and demeaning.
To qualify, trans people do not need to have had surgery, but must have lived for two years in their preferred gender.
But some trans people believe they should be allowed to self-declare the gender in which they choose to live, without the need for medical evidence or proof.
"The moral panic we see at the moment isn't really about the gender recognition act, in the same way the really undignified public discourse around gay marriage wasn't really about gay marriage - it is just an excuse for people to vent really ugly homophobia," Ms Lees argues.
"All we are hearing about is all of people's imagined fears about how it will be the end of the world if we make it easier for you and I to change our birth certificates."
Journalist Ella Whelan believes there is no need for legislation on the issue.
"It's playing into this idea that if you don't believe a trans woman is a woman, or a trans man is a man, then you are not only the wrong kind of person in society and bigoted, but also you're going against the law and what the law is," she says.
"And that should be up for debate."
The Gender Identity Research & Education Society (GIRES) estimates that about 1% of the British population are gender-nonconforming to some degree.
Meanwhile, LGBT charity Stonewall found that 41% of trans people have experienced a hate crime because of their gender identity in the past 12 months.
The Equality Act 2010 protects trans people from discrimination and says anyone who identifies themselves as a woman can use single-sex facilities.
It does however allow service providers - including hospitals and prisons - to refuse a trans person access to single-sex services if that is seen as detrimental to others.
Dr Timothy Hildebrandt, from the London School of Economics, says the UK government doesn't "see" transgender people, because they do not fit into the way services, such as healthcare and prisons, have always been divided between the genders.
"The difficulty then with gender identity discussions, there is simply no way for governments to really understand them because they don't fit within the general framework of dividing the general population," he says.
For her part, Ms Whelan does not think the law needs updating.
"You're saying we are going to have a completely new system in a prison to placate the personal situations of a few trans people; well there might be some questions about the needs to do that and it might not be necessary," she says.
The issue of trans women in politics is also making waves.
A group of Labour party members have set up a crowd-funder to try and stop the party allowing self-identifying trans-women being selected for all-women shortlists for their candidates.
Dr Heather Peto, is a trans woman who was selected by her local Labour party to be on the all-women shortlist for Rushcliffe and Melton.
She says: "I find it incredible that a small group of women can say who and who shouldn't represent them - if the membership wants to select me, they'll select me and if they don't they don't.
"It is up to them to decide how many people stand and I'm as valid a woman as any other woman - it is just that I've had a bit of a different journey getting there."
The Labour Party said all-women shortlists are, and always have been, open to all women "which of course includes trans women".
Meanwhile, some say the strength of feeling that these issues create means that even discussing them is difficult.
Rebecca Root, an actor who transitioned in 2003, says: "But why should there be a debate at all: it's that person's life."
"They're not really doing anything to harm anybody else by saying I'm female or male of whatever," she adds.
Watch the BBC's Victoria Derbyshire programme on weekdays between 09:00 and 11:00 on BBC Two and the BBC News Channel.