The judge heading the Grenfell Tower inquiry should stand down because those affected need "somebody we can trust", the local MP for Kensington has said.
Labour's Emma Dent Coad said Sir Martin Moore-Bick was "a technocrat" who lacked "credibility" with victims.
On Monday, lawyers representing some of the families called for him to quit.
Labour leader Jeremy Corbyn said Sir Martin should "listen to residents", but minister David Lidington said he had "complete confidence" in him.
The Tory MP and Lord Chancellor said Sir Martin would lead the inquiry "with impartiality and a determination to get to the truth and see justice done".
Mayor of London Sadiq Khan warned the retired Court of Appeal judge must urgently improve relations with local residents.
The fire on 14 June is thought to have killed at least 80 people, although police say the final toll will not be known until at least the end of the year.
Meanwhile, official figures from the Grenfell Response Team show 139 offers of accommodation have been made to families left homeless, but just nine have been accepted.
In the aftermath of the tragedy, the prime minister promised that 158 families would be offered a good quality temporary home within three weeks - a target the response team said has now been met.
The remaining 19 families do not want to be contacted, or are out of the country, it added.
One tenant from the 10th floor of Grenfell Tower, who only gave his name as Antonio, is among those who turned down the offer of temporary accommodation.
"We want to move to a permanent accommodation so we can remake it and then we can call it home," he told BBC Radio 5 live.
Speaking after his appointment as chairman of the inquiry, Sir Martin said he understood the "desire of local people for justice" following the disaster, but warned he was "doubtful" the process would be as wide-ranging as some residents hoped.
On Monday, however, a source told the BBC he was prepared to be "open-minded" in his "very broad" inquiry.
Although the remit will be decided by the prime minister, the source said he would consider in detail whether the nature of the building regulations contributed to the fire.
Despite that attempted reassurance, Ms Dent Coad said she had spoken to hundreds of people affected by the fire who were unhappy with his appointment.
"We need somebody who can do the detail but we need somebody who can actually understand human beings as well and what they've been through..." she told BBC Radio 4's Today programme.
"I don't think he should do it. I don't think there will be any credibility.
"Some people are saying they are not going to co-operate with it, so it's not going to work."
Her call for Sir Martin to stand down has been backed by shadow fire services minister Chris Williamson.
"I think it's important that we listen to the survivors... local people are saying they don't have faith in him," he told BBC Radio Derby.
He said he agreed with shadow chancellor John McDonnell's comment that the victims were "murdered by political decisions".
"If you look at decisions taken over the last three or four decades, it's all about unbridled market economics - essentially deregulation - using public services as a cash-cow and cutting corners," he said.
On Sunday, Labour MP David Lammy said a "white, upper-middle class man" who had "never" visited a tower block housing estate should not have been appointed.
Grenfell residents have also questioned whether Sir Martin's background in commercial law is appropriate.
And they have been angered by his decision to allow Kensington and Chelsea Council - which was criticised for its slow and ineffective response to the disaster - to contribute to the inquiry.
Elsewhere, Met Police Commander Stuart Cundy and Westminster coroner Fiona Wilcox - who will lead will the inquests of the victims of the fire - will hold a private meeting with relatives later.
Who is Sir Martin?
- Educated at Christ's College, Cambridge
- Specialised in commercial law, involving maritime and land transport disputes
- Spent more than 20 years as a judge of the Commercial Court and Court of Appeal
- Retired in 2016
- In November 2014, he ruled Westminster City Council could rehouse a single mother-of-five more than 50 miles away, in Milton Keynes.
- The decision was overturned by the Supreme Court in April 2015.