A victim of historical child sexual abuse has launched a legal challenge to the choice of Fiona Woolf as the chair of the inquiry investigating the issue.
A judicial review application, seen by the BBC, claims she is not impartial, has no relevant expertise and may not have time to discharge her duties.
Labour wants Mrs Woolf to meet abuse victims amid concerns over her links to former Home Secretary Lord Brittan.
Downing Street said it had "full confidence" in her doing the job.
BBC assistant political editor Norman Smith said he believed the government would do everything it could to "cling onto" Mrs Woolf given that her predecessor in the role had already stood down and the inquiry was being asked to produce an interim report by the end of March.
Questions have been asked about Mrs Woolf's suitability for the role given that she has no experience in child protection or family law and on Tuesday revealed that she had dined with Lord Brittan - home secretary in 1984 when ministers were handed a dossier on alleged high-profile paedophiles - five times since 2008.
The legal challenge to her appointment could be heard before the end of the year.
Judicial review is used to challenge the decisions of government and other organisations based on the law. Most applications do not get past the initial stage and are rejected by a judge.
'Lack of confidence'
The application comes amid calls from a number of MPs from different parties for her to consider her position.
Lib Dem MP John Hemming said Mrs Woolf's personal contacts "would give no confidence to the victims that have been ignored for many years", while Conservative Nadine Dorries tweeted that Mrs Woolf "no longer" had full public and victim confidence.
For Labour, shadow energy secretary Caroline Flint told the BBC's Daily Politics that it was "really difficult for her (Mrs Woolf) to stay" if survivors of child abuse did not support her.
The opposition is urging Mrs Woolf and Home Secretary Theresa May to meet survivors of abuse immediately to address their concerns.
But Cabinet Office minister Francis Maude appealed for critics of Mrs Woolf to "step back here", telling BBC Radio 4's World at One: "This is really important and really matters. It has to be done right.
"Fiona Woolf seems to be a really admirable person in many ways... She's not doing this on her own. She's tried to be completely meticulous about disclosing all these matters. I hope we can deal with things in a grown-up way."
But Andi Lavery, who describes himself as a "survivor of abuse" at a school in the 1980s, said he had "zero confidence" in the inquiry, suggesting it had "no investigatory powers" and Mrs Woolf appeared "weak as water" when questioned by MPs on Tuesday.
Peter McKelvie, a former child protection manager whose allegations about child abuse led to a 2012 police inquiry, told the BBC's Newsnight it was "quite an insult" that someone with links to establishment figures had been appointed to lead the inquiry.
He said trust in the process was a "major element" for abuse victims, but survivors he had spoken to had "no trust in the whole process".
"It's a process that survivors I'm talking to say they really don't want to be part of. I really don't think they will cooperate with this particular process at all," he said.
Alison Millar, whose law firm Leigh Day is representing victims of abuse, said: "This is not really about Fiona Woolf's ability or her integrity, this is about her independence and her ability to lead this inquiry in a way that is credible to the survivors of abuse.
"Somebody who seems to be on dinner party terms with a senior political figure whose knowledge this inquiry will be scrutinising is somebody who, from the perspective of my clients, does not have the necessary independence."
But Sharon Evans, a member of the inquiry panel, said it had "got off to a difficult start" but she was confident it was representative, with a QC and investigator from Her Majesty's Inspectors of Constabulary as well as people, like herself, with personal experience of abuse.
"There are nine people on the panel with an enormous background and expertise in this job," Ms Evans, chief of the Dotcom Children's Foundation, told Today.
"There are victims on this panel and we are determined to get to the bottom of this. The fact that we are part of this will give people confidence and we will listen."
And Downing Street restated its support, saying it was sure that the panel "would look at all the ways" they could reassure victims of abuse that the inquiry was independent.
Mrs Woolf, who is currently Lord Mayor of London, confirmed on Tuesday she lived in the same London street as Lord Brittan and had invited him and his wife to dinner on three occasions, and had two dinners at their house between 2008 and 2012.
But appearing before the home affairs committee of MPs on Tuesday, she said the Conservative peer was "one of thousands of people" she knew and not a "close associate".
Mrs Woolf was appointed after the government's original choice, Baroness Butler-Sloss, stepped down after victims' families expressed concerns about her independence.
Her late brother Lord Havers was attorney general at the time allegations about high-profile paedophiles were passed to ministers by the late Conservative MP Geoffrey Dickens.
The dossier was considered by Home Office officials in 1984 and handed to police but no action was taken and the information has since disappeared.
Lord Brittan has insisted that proper procedures were followed.
The inquiry, triggered by allegations of child sex abuse by public figures working in the media, politics and other walks of life, will examine the conduct of almost all major UK institutions.