Boris Johnson says he has been "a model of restraint" when it comes to language around the Brexit debate.
The PM was accused of dismissing abuse fears of female MPs as "humbug" during a heated Commons debate this week.
Mr Johnson said there had been a "misunderstanding" over his intention - which he apologised for.
But he claimed there was a "cloud of indignation" around the use of terms like "surrender" to distract from MPs' desire to frustrate Brexit.
Labour's Angela Rayner said Mr Johnson should be "absolutely, utterly ashamed of himself".
The shadow education secretary said the PM had "a direct strategy to divide our country", which she called "really damaging", but added that MPs on both sides of the Commons needed to "dial down that language and act responsibly".
Mr Johnson spoke to the BBC's Andrew Marr ahead of the start of the Conservative Party conference in Manchester.
The annual gathering follows a turbulent week for the PM - facing stormy scenes in Westminster after a ruling by the Supreme Court that his suspension of Parliament was unlawful.
Rows later broke out over Mr Johnson's Commons conduct - and that of his Attorney General Geoffrey Cox - and questions are also continuing over his links with US businesswoman Jennifer Arcuri.
But he will use the conference to try to focus on delivering Brexit and making funding promises for public services.
On the opening day, ministers have promised to spend billions on hospital projects across England in the next decade.
But opposition MPs back in London could stage a no-confidence vote in the government some time this week.
No 10 failed to secure a recess for the conference, meaning Tory MPs could face travelling between Manchester and Westminster for crucial votes while it takes place.
Labour leader Jeremy Corbyn is also expected to bring opposition leaders together at a meeting on Monday to plan their next steps to avert a no-deal Brexit.
Mr Corbyn has said he would be ready to become a caretaker PM if Mr Johnson was forced from No 10.
Greater Manchester Police said they were expecting tens of thousands of protesters to attend two rallies on Sunday against Brexit and austerity.
A small number of demonstrators also gathered outside the conference venue on Saturday night when Mr Johnson arrived with his partner, Carrie Symonds.
Speaking to Andrew Marr, Mr Johnson defended his decision to repeatedly refer to the Benn Act - which is designed to force the PM to seek an extension rather than lead the country into a no-deal Brexit - as "the surrender bill".
Last week, Labour's Paula Sheriff said threats made to her and other MPs often quoted his "dangerous" language of "surrender" and "betrayal".
She also referred to the murder of Labour MP Jo Cox in 2016, but Mr Johnson replied: "I've never heard such humbug in all my life."
The PM said any such threats were "deplorable" but he did not regret "using the word surrender to describe the surrender act".
"Military metaphors are old, standard, Parliamentary terms," he continued.
"I think everybody should calm down," he said, but asked whether that included him, he added: "I think I've been the model of restraint."
The Jo Cox Foundation - set up in memory of the murdered MP - has called on all political parties to agree to a code of conduct to help protect MPs.
Cardinal Vincent Nichols, the UK's most senior Catholic clergyman, warned some of the language being used could encourage violent extremists.
He told BBC Radio 4's Sunday programme political leaders had a particular responsibility to be respectful in public discourse.
Commons Speaker John Bercow will hold an emergency meeting on Monday with party leaders over the use of "inflammatory language" in Parliament.
What is the Benn Act?
When Mr Johnson talks about the "surrender bill", he is referring to the European Union (Withdrawal) (No. 2) Act, also known as the Benn Act after Labour MP Hilary Benn, who introduced the legislation to the Commons.
The act - which became law earlier this month - stipulates the prime minister will have until 19 October to either pass a deal in Parliament or get MPs to approve a no-deal Brexit.
Once this deadline has passed, he will have to request an extension to the UK's departure date to 31 January 2020 from the EU.
If the EU responds by proposing a different date, the PM will have two days to accept that proposal. But during this two-day period, MPs - not the government - will have the opportunity to reject the EU's date.
On Brexit, Mr Johnson said he still thought there was a "good chance" of getting a deal with the EU and the government was "working incredibly hard".
Success depended on "the common sense of our EU partners", he argued, but said talks were "not being helped" by the Benn Act.
"If they suspect or think there is a realistic chance the UK can be kept in [the EU], that clearly takes away a lot of our negotiating freedom."
The PM insisted that "of course" the UK could still leave without a deal on 31 October despite the act - although he would not be drawn on how that could happen.
Mr Johnson also ruled out bringing back the withdrawal agreement negotiated between Theresa May and the EU, and dismissed suggestions he could resign rather than ask for an extension.
There's no sign whatsoever of Boris Johnson retreating from his strategy - the approach he's taken since he moved into Downing Street. And as far as many people in his team, and the wider party, are concerned their only real path to success is to stick to trying get Brexit done by 31 October.
The tricky thing is, though, that it looks very unlikely - maybe even impossible. And there are plenty of people in Westminster who say they're prepared to do everything necessary to stop Mr Johnson doing what he wants to do - taking the UK out without a deal on that date if one isn't agreed.
Party conferences are pretty much always like being in a parallel universe, but this week is going to be something else.
Parliament will still be sitting while the Conservatives are gathering here in Manchester and it could certainly be a surreal and bumpy few days.
Meanwhile, the Sunday Times has fresh allegations about the American businesswoman at the centre of claims about her links to the prime minister.
The paper claims Jennifer Arcuri told four friends that she had an affair with Mr Johnson while he was mayor of London.
On Friday, the Greater London Authority (GLA) referred Mr Johnson to the police watchdog over allegations Ms Arcuri, a US technology entrepreneur, received favourable treatment because of her friendship with him.
She joined trade missions led by Mr Johnson while he was mayor and her company received several thousand pounds in sponsorship grants.
The Independent Office for Police Conduct will now consider whether there are grounds to investigate the prime minister for the criminal offence of misconduct in public office.
But Mr Johnson insisted he had "no interest to declare" when pressed by Andrew Marr over whether he made his personal relationship with her known at the time.
"Everything was was done with full propriety," he added.
The PM also appeared to claim the story was politically motivated, saying someone in his position "expects a lot of shot and shell".
As well as promises on hospitals, the government is also announcing a new approach to NHS mental health treatment to be trialled in 12 areas of England - with housing and job support alongside psychological help.
Chancellor Sajid Javid is expected to speak on Monday and Home Secretary Priti Patel - talking about tackling crime - on Tuesday.
Animal welfare policies, environmental plans - including a proposed £1bn fund to boost the electric motor industry and a pledge to plant one million new trees - are on the agenda.