Prime Minister David Cameron says he is "completely fed up" about the release on bail of Abu Qatada after the Muslim cleric won his deportation appeal.
Abu Qatada was earlier freed from prison after a UK court ruled he might not get a fair trial if deported to Jordan to face bomb plot charges.
Mr Cameron said ministers had "moved heaven and earth" to try to deport him and would continue to do so.
Labour said people would be "horrified" and urged ministers to act quickly.
Mr Cameron told BBC News: "I am completely fed up with the fact this man is still at large in our country, he has no right to be there, we believe he's a threat to our country
"We have moved heaven and earth to try and comply with every single dot and comma of every single convention to get him out of this country.
"It's extremely frustrating and I share the British people's frustration at the situation we find ourselves in," Mr Cameron said.
He said that, as well an appeal, the government would "do everything we can to make sure that we do have the power to expel and deport people from our country who have no right to be there and mean us harm".
BBC legal correspondent Clive Coleman said Home Office lawyers had 21 days to find a point of law to take to the Court of Appeal.
If the appeal was knocked out, Abu Qatada's stringent bail conditions would "simply fall away", he said, leaving the home secretary with a stark choice.
Theresa May could do the "politically unthinkable" and release Abu Qatada or she could impose a new Terrorism Prevention and Investigation Measures (TPim) control order which itself could be subject to legal challenge.
Faced with this difficult problem, there would also be furious diplomatic efforts to beef up assurances from the Jordanian government, our correspondent added.
Abu Qatada - whose real name is Omar Othman - arrived back at his home in London earlier after he was released from Long Lartin prison, in Worcestershire. He has spent most of the last 10 years in custody.
Earlier this year, judges at the European Court in Strasbourg ruled the cleric would not face ill-treatment if returned to Jordan, citing assurances outlined in a UK-Jordan agreement.
But the judge did not believe he would get a fair trial because a Jordanian court could use evidence against Abu Qatada that had been obtained from the torture of others.
On Monday, the Special Immigration Appeals Commission chairman Mr Justice Mitting ruled he was not satisfied Abu Qatada would be tried fairly.
Judges say there is a real risk that the preacher's retrial in Jordan would be unfair because it would include incriminating statements made by men who were tortured by the secret police.
Jordan's acting information minister Nayef al-Fayez told the BBC his government shared UK authorities' disappointment at the Special Immigration Appeals Commission (Siac) ruling on Monday.
Shadow home secretary Yvette Cooper said people would be "horrified that Abu Qatada is now out on Britain's streets rather than on a plane" and urged ministers to go to Jordan "straight away to discuss what additional action would get this sorted out".
Human rights lawyer Julian Knowles, meanwhile, said the case would bring "another year's worth of UK litigation at least".
"And then if Abu Qatada is the loser at the end of the domestic phase, he can then go back to the European Court," he said.
On the question of why the Palestinian-born Jordanian had never been prosecuted in the UK, Lib Dem peer Lord Macdonald - director of public prosecutions from 2003 to 2008 - told BBC News he had never been shown any evidence to support a criminal prosecution.
The bail conditions imposed by Mr Justice Mitting on Abu Qatada include being allowed out of his house only between 08:00 and 16:00, having to wear an electronic tag, and being restricted in whom he meets.
Abu Qatada faces a retrial in Jordan for allegedly conspiring to cause explosions on Western and Israeli targets in 1998 and 1999. He was found guilty of terrorism offences in his absence in Jordan in 1999.
Security chiefs believe he played a key ideological role in spreading support for suicide bombings.