Foreign Secretary Dominic Raab was called a "coward" by the friends and family of Harry Dunn as they were left outside constituency hustings.
Mr Dunn, 19, died in a crash in Northamptonshire in August that led to the suspect leaving the UK claiming diplomatic immunity.
His family had hoped to put pressure on Mr Raab at the event in Surrey.
Mr Raab said he felt a "constructive conversation" would not have been possible due to crowds there.
A member of staff at the church where the hustings were held said they were kept outside due to fire safety.
Mr Dunn's parents have begun legal action against the Foreign and Commonwealth Office (FCO), alleging the granting of immunity by Mr Raab was "wrong in law".
But the FCO has said it will "seek costs" for any judicial review brought and argue the family has not found "any reasonably arguable ground of legal challenge".
It said an allegation that the foreign secretary had "misused and/or abused his power" was "entirely without foundation".
Signs calling for Harry's family and friends to be allowed to enter were held up against the East Molesey Methodist Church door, which was monitored by staff.
There were chants of "let us in" from more than 50 people who were left outside on Monday night.
As Mr Raab left the church in an official car, a crowd booed and branded him a "coward".
Mr Raab told the BBC: "I'm happy to see the family whenever they want to, but doing it outside in those circumstances… I don't think was the right thing to do, it's not like we'd be able to have a constructive conversation."
He reiterated his sympathy for the Dunn family and said he would "like to see justice done" but that he had "no control over the case".
Harry's father Tim Dunn said he had hoped to call on voters to back another candidate to unseat Mr Raab at the Esher and Walton constituency hustings.
Asked why he thought the foreign secretary did not speak to him, Mr Dunn said: "I think he doesn't want to give me the answers I need.
"There's things going on, we're not quiet about it, we think the CPS (Crown Prosecution Service) and the FCO are talking and delaying tactics."
Mr Dunn was fatally injured on 27 August, when his motorbike was in collision with a car owned by Anne Sacoolas outside RAF Croughton, where her husband Jonathan was an intelligence officer.
Mrs Sacoolas, 42, left the UK claiming diplomatic immunity but the family are seeking a judicial review of that decision.
During the hustings, Rebecca Little, who lives in Cobham, said she asked about Mr Dunn's case on behalf of his family.
"There were several security detail preventing the family from coming in but they have a right to enter the hustings," she said.