Baroness Butler-Sloss has cautioned against giving victims too much influence over who runs the planned inquiry into historical child abuse.
The retired judge, who stepped down as head of the public inquiry, said there could be "real problems" if they were to decide who is its eventual chair.
She also told BBC Radio 4 she has "enormous sympathy" for the victims.
The home secretary will decide who heads the inquiry, but the government says she wants to hear victims' views.
A Home Office spokesman said Home Secretary Theresa May is "absolutely committed" the inquiry has the "confidence of survivors... to ensure the right person is appointed".
The inquiry, sparked by claims of paedophiles operating in Westminster in the 1980s, is set to investigate whether "public bodies and other non-state institutions have taken seriously their duty of care to protect children from sexual abuse in England and Wales".
The panel has started work but has no-one to lead it after its first two nominations resigned. Mrs May, who is still considering the format of the inquiry, has told the current panel it might be disbanded.
Dozens of survivors have called for the government to scrap the current inquiry and replace it with a more powerful body.
Lady Butler-Sloss stood down earlier this year amid claims she faced a conflict of interest because her late brother, Sir Michael Havers, was attorney general at the time of some of the alleged abuse.
Speaking to the Today programme, of which she was guest editor on Wednesday, Lady Butler-Sloss said "there has to be a victim voice on the panel" but the survivors should not be able to chair it themselves or choose who fills the position.
She said: "You need someone who knows how to run things and if you get someone from an obscure background, with no background of establishment, they'll find it very difficult and may not be able actually to produce the goods."
Lucy Duckworth, the chair of Minister and Clergy Sexual Abuse Survivors, said Lady Butler-Sloss's comments were "shockingly naive" and "patronising to victims".
Lady Butler-Sloss said establishment figures had covered up abuse in the past.
She said this was "partly because people did not really recognise the seriousness of child abuse and they did not think it was so important, and it was important to protect members of the establishment".
She said she had no regrets about accepting the role as the first chair of the inquiry and had thought it was her "duty" to do so.
When asked if the troubled inquiry would ever "get off the ground", Lady Butler-Sloss replied: "I don't know."
The Liberal Democrat MP Tessa Munt told Radio 4's PM programme the delay to the inquiry was "frustrating", but she believed the home secretary was determined to see it take place.
Ms Munt, who earlier this year revealed she suffered from child abuse, was one of the MPs who first wrote to the home secretary asking for an independent inquiry.
She said she was "quite convinced we will uncover some terrible things... but it's going to happen".
She reiterated her belief that there would be an advantage in appointing someone from a Commonwealth country who would "understand our culture but is not close to the establishment" to head the inquiry.
Fiona Woolf, who has been made a dame in the New Year Honours list, also stood down amid questions over her links to former Home Secretary Lord Brittan.
Speaking on Today, Lady Butler-Sloss said criticism of the honour awarded to Dame Fiona was "very unfair".
"She was Lord Mayor of London, she is only the second woman ever to be Lord Mayor of London," she said.
"The very least that the honours system could do would be to honour a woman who has got such a distinguished post.
"Unfortunately she had, like myself, a brief period where she had agreed... to become chairman. And because she happened to know Leon Brittan, she was unacceptable to the survivors and therefore she stood down."
Lord Brittan could be called to give evidence to the inquiry about a dossier on alleged high-profile paedophiles that was handed to ministers in 1984.