There have been chaotic scenes outside the UN's Sri Lanka offices, where protesters are demanding the UN end investigations into alleged war crimes.
Most UN staff managed to leave after police tried to break the blockade in Colombo, before being ordered to leave themselves by the government.
Several senior staff, however, remain inside and the protests, led by a government minister are continuing.
Sri Lanka says an inquiry is not needed and denies troops committed war crimes.
The government has refused to grant visas to the UN advisory panel's three members, saying the investigation violates its sovereignty.
Speaking from New York, a UN spokesman told the BBC the organisation was doing all it could to ensure the safety of its Colombo staff.
"The UN has registered its strong objections to protests organised outside UN offices... which prevented staff and visitors from entering or leaving the premises," spokesman Farhan Haq told BBC Sinhala.
"While respecting the right of citizens to demonstrate peacefully, preventing access to UN offices hinders the vital work being carried out by the United Nations each day to help the people of Sri Lanka.
"The government has provided assurances for the safety and security of our staff and for their full access to their offices. We will be closely monitoring developments and trust these commitments will be honoured."
There have been consistent allegations that both the army - and Tamil Tigers rebels who troops routed last year - committed crimes at the end of the war.
About 7,000 civilians died in the last five months of the war, according to the UN.
It says the panel, announced last month, will report back within four months and will advise on how to deal with alleged perpetrators of abuses.
Hundreds of flag-waving protesters, led by Housing Minister Wimal Weerawansa, descended on the UN offices on Tuesday. They burned an effigy of UN Secretary-General Ban Ki-moon outside the buildings, the BBC's Charles Haviland reported from the scene.
They also threatened to go on hunger strike to press their demands.
Police arrived in the afternoon and there were clashes with protesters when they attempted to break the siege. Police managed to escort several vehicles of UN staff out of the compound.
An angry Mr Weerawansa then called up the president's brother, Defence Secretary Gotabhaya Rajapaksa, and put him on the speaker phone, our correspondent reports.
Mr Rajapaksa ordered the police to leave and soon hundreds of police officers obeyed. Scores of protesters still remain outside the UN office, our correspondent says.
The protesters, many of them Buddhist monks, say their action will continue until the UN disbands the panel. The UN said there were no plans to disband it.
Mr Weerawansa told the BBC that "patriotic Sri Lankans" did not support the three-member panel.
Meanwhile, the detained ex-army chief Sarath Fonseka, who led the security forces in fighting the Tamil Tigers, has said he is not afraid to face the UN panel.
"As the then army commander I can candidly say the war was waged in line with international covenants and conventions. I fully support the military in this case," Sri Lanka's Daily Mirror newspaper quoted him as saying.
There has been much international concern over the conduct of both sides in the latter stages of the war, which ended in May 2009.
Sri Lanka says it will hold its own internal inquiry, but the exact terms of reference are not clear.
International human rights groups are sceptical about the ability of the government to investigate claims impartially. They are demanding an independent investigation.
On Monday, the European Union announced it was withdrawing Sri Lanka's preferential trade access to EU markets after it failed to improve its human rights record.