Victims' groups have told Home Office officials they are "unanimous" in the view that the head of an inquiry into historic child sex abuse should resign.
The inquiry would be a "dead duck" if Fiona Woolf remained chairwoman, Peter Saunders from National Association for People Abused in Childhood said.
Victims have called for her to step down because of her social links to ex-Home Secretary Lord Brittan.
A spokesman said PM David Cameron was "absolutely clear" she can do the job.
The inquiry will look at whether public bodies and other institutions did enough to protect children from sexual abuse from 1970 to the present day.
Speaking after a meeting between victims' groups and officials from the inquiry, campaigners called for Mrs Woolf to be replaced and for a statutory inquiry to be set up with powers to seize documents and compel witnesses to give evidence.
A statement from Alison Miller, of Leigh Day Solicitors, said child abuse survivors' representatives were "unanimous" that Mrs Woolf was unsuitable to lead the inquiry.
Mr Saunders, from NAPAC, said Mrs Woolf was someone "we would not be able to work with", adding that it was "essential" it became a statutory inquiry.
Dr Liz Davies, who was a social worker in Islington and acted as a whistleblower there, said victims from the London borough had adopted a policy of "absolute non co-operation" with the inquiry as it stands.
"Something really has to change," she said.
The NSPCC was also at the meeting in London, which began at about 10:30 GMT. The charity has declined to give explicit backing to Mrs Woolf, a corporate lawyer.
Home Affairs Select Committee chairman Keith Vaz told the BBC he had written to Mrs Woolf asking her to return to the committee next week to "clarify outstanding points".
It comes after the first person appointed to lead the inquiry - Baroness Butler-Sloss - stepped down in July after concerns were raised about the fact that her late brother was attorney general during the 1980s.
Ross Hawkins, BBC political correspondent
The voices of abuse survivors were always going to be vital.
The phone hacking inquiry proved that victims have a moral authority like no other and their opinions will be heard.
Yet survivors and their representatives at today's meeting said they had no contact from the Home Office or the inquiry until last week.
Survivors are concerned about the way the inquiry has been established, its powers and its terms of reference.
One survivor - Phil Johnson - said that when he asked for his travel expenses to be repaid, he was told that could not be guaranteed.
So a victim of abuse summoned to meet officials had to leave his Eastbourne home after rush hour - to avoid the highest train fares - then run from the station to get to the meeting in time.
He said: "I think the victims are being taken for granted."
A victim of historical child sexual abuse has already launched a legal challenge to Mrs Woolf's appointment, claiming she is not impartial, has no relevant expertise and may not have time to discharge her duties.
BBC legal correspondent Clive Coleman said senior or former judges were the obvious choices as chair of long inquiries as they were used to hearing and analysing vast amounts of testimony from multiple witnesses.
However, our correspondent said victims and survivors must have confidence the chair is "divorced from the contentious subject matter of the inquiry and key people who may figure in it".
Earlier this month Mrs Woolf, who is Lord Mayor of London, disclosed that she lived in the same street as Lord Brittan and had dinner with him five times between 2008 and 2012 - but said he was not a "close associate".
Lord Brittan may be called to give evidence to the inquiry. He denies any wrongdoing in the way the "dossier" on alleged high-profile paedophiles was handled in the 1980s.
On Thursday, MP Mr Vaz said letters from Mrs Woolf showed her appointment was "chaotic". He said a letter from Mrs Woolf about her links with Lord Brittan was re-written seven times until the final version gave a "sense of greater detachment".
Former Labour home secretary David Blunkett told BBC Radio 4's The World at One there was now a "very substantial cloud" over whether Mrs Woolf could continue as head of the inquiry.
He said revelations about the way the letters had been edited put a "very different complexion" on the situation.
The letters: What they said and when
Mr Cameron's official spokesman said the prime minister's view that Mrs Woolf should lead the inquiry "has not changed".
"The prime minister is absolutely clear he thinks she can do this job with integrity and impartiality," he added.
Labour's shadow home secretary, Yvette Cooper, said Home Secretary Theresa May had "totally failed" to get the inquiry going.
Abuse inquiry: How we got here
1 July - MP Simon Danczuk calls on former Home Secretary Leon Brittan to say what he knew about paedophile allegations passed to him in the 1980s
7 July - Government announces independent inquiry into the way public bodies investigated and handled child sex abuse claims. Baroness Butler-Sloss chosen as head
9 July - Baroness Butler-Sloss (pictured) faces calls to quit because her late brother, Sir Michael Havers, was attorney general in the 1980s
14 July - She stands down, saying she is "not the right person" for the job
5 September - Lord Mayor of London Fiona Woolf named the new head of the inquiry
11 October - Mrs Woolf discloses she had five dinners with Lord Brittan from 2008-12
22 October - Abuse victim launches legal challenge against Mrs Woolf leading the inquiry, amid growing calls for her resignation