Foreign Secretary Boris Johnson has said Libyan city Sirte could be the new Dubai, adding, "all they have to do is clear the dead bodies away".
His comments at a Conservative fringe meeting sparked anger, with a number of Tory MPs calling for his sacking and Labour labelling him "crass and cruel".
Mr Johnson claimed his critics had "no knowledge nor understanding of Libya".
A Downing Street source said it was not an "appropriate choice of words" but the PM regarded the matter as closed.
"I look at Libya, it's an incredible country," Mr Johnson told the meeting.
"Bone-white sands, beautiful sea, Caesar's Palace, obviously, you know, the real one.
"Incredible place. It's got a real potential and brilliant young people who want to do all sorts of tech.
"There's a group of UK business people, actually, some wonderful guys who want to invest in Sirte on the coast, near where Gaddafi was captured and executed as some of you may have seen.
"They have got a brilliant vision to turn Sirte into the next Dubai.
"The only thing they have got to do is clear the dead bodies away," he said, before laughing.
The host of the conference fringe event, Legatum Institute chief executive Baroness Stroud, stepped in to say "next question", as the foreign secretary continued to speak.
The coastal city of Sirte is the former stronghold of so-called Islamic State, or Daesh, and recently the scene of fierce fighting.
Reacting on Twitter, Ms Allen said: "100% unacceptable from anyone, let alone foreign sec. Boris must be sacked for this. He does not represent my party."
Conservative MP Sarah Wollaston called on Mr Johnson to apologise and urged him to "consider his position", adding that the comments were "crass, poorly judged and grossly insensitive - and this from the person who is representing us on the world stage. I think they were really disappointing."
And Justice Minister Philip Lee tweeted that "anyone decent" would condemn the comments.
But fellow Tory MP Nadine Dorries tweeted that "the campaign by Remain MPs on here calling for Boris to resign" was "co-ordinated and mendacious".
Mr Johnson defended his remarks, adding on Twitter that he had been making a point about the need for optimism in Libya, after a recent visit to the country.
"The reality there is that the clearing of corpses of Daesh fighters has been made much more difficult by IEDs and booby traps," he tweeted.
"That's why Britain is playing a key role in reconstruction and why I have visited Libya twice this year in support."
But Damian Green, the first secretary of state, told BBC 5 live he believed Mr Johnson's remarks were unacceptable, adding: "It was not a sensitive use of language. As I say, we all need to be sensitive in our use of language, particularly in situations like that."
Labour's shadow foreign secretary, Emily Thornberry, said: "It is less than a year since Sirte was finally captured from Daesh by the Libyan Government of National Accord, a battle in which hundreds of government soldiers were killed and thousands of civilians were caught in the crossfire, the second time in five years that the city had seen massive loss of life as a result of the Libyan civil war.
"For Boris Johnson to treat those deaths as a joke - a mere inconvenience before UK business people can turn the city into a beach resort - is unbelievably crass, callous and cruel.
"If these words came from the business people themselves, it would be considered offensive enough, but for them to come from the foreign secretary is simply a disgrace.
"There comes a time when the buffoonery needs to stop, because if Boris Johnson thinks the bodies of those brave government soldiers and innocent civilians killed in Sirte are a suitable subject for throwaway humour, he does not belong in the office of foreign secretary."
Lib Dem deputy leader Jo Swinson said the "unbelievably crass and insensitive comment" was further proof Mr Johnson was "not up" to a job for which diplomacy was "a basic requirement".