Boris Johnson has insisted allegations about his personal conduct are not overshadowing the Tory conference.
Journalist Charlotte Edwardes has accused the PM of touching her thigh, and that of another woman, at a lunch in 1999, which he denies.
Rumours were circulating at conference that Mary Wakefield - who is married to the PM's chief adviser - was the second woman, but she has rejected that.
The PM said people wanted to hear about his plans to "improve their lives".
The row erupted after Ms Edwardes wrote a column in the Sunday Times on the eve of the Conservative conference in Manchester, describing the alleged incident.
The PM was already facing questions over his ties to a US businesswoman, Jennifer Arcuri, during his time as London mayor - he insists he acted with "full propriety".
The Conservatives are trying to focus this week on their key conference message - "Get Brexit Done" - and a raft of policy pledges.
In her first column for the paper, Ms Edwardes said the incident took place in 1999. She said she was seated on Mr Johnson's right at the lunch, held at the Spectator magazine's offices - he was editor of the magazine at the time.
"More wine is poured; more wine is drunk. Under the table I feel Johnson's hand on my thigh. He gives it a squeeze," she wrote.
"His hand is high up my leg and he has enough inner flesh beneath his fingers to make me sit suddenly upright."
On Sunday evening, No 10 released a statement calling the claims "untrue", but Ms Edwardes later tweeted: "If the prime minister doesn't recollect the incident then clearly I have a better memory than he does."
Asked on Monday if the incident had taken place, Mr Johnson said: "No."
When it was put to him that the row was overshadowing everything else at the conference, he replied "not at all".
"I think what the public want to hear is what we are doing to bring the country together and get on with improving their lives," he added.
Ms Edwardes said another woman at the lunch later told her Mr Johnson had done the same to her.
Spectator magazine commissioning editor Ms Wakefield, who is married to the prime minister's adviser Dominic Cummings, issued a statement to say she was "not the woman referred to in Charlotte Edwardes's column".
"Boris was a good boss and nothing like this ever happened to me. Nor has Charlotte, who I like and admire, ever discussed the incident with me."
Earlier, ex-Tory minister Justine Greening said Ms Edwardes' story was "deeply concerning" and raised questions "about character and integrity of people in public life and what standards the electorate have a right to expect".
Chancellor Sajid Javid, however, backed the PM, saying: "I have full faith in the prime minister and I don't doubt that and what he has said for a second."
On Sunday, Health Secretary Matt Hancock said "these issues are incredibly important".
He said he knew Ms Edwardes well and knew her to be "trustworthy".
Former minister Amber Rudd - who quit the Conservative Party over its handling of Brexit earlier this month - tweeted that she agreed with Mr Hancock's conclusion.
But Chief Whip Mark Spencer told BBC Nottingham on Monday: "If the prime minister says it didn't happen, it didn't happen."
He added: "If I were a victim of some sort of assault, I would go to the police straight away... if that's what happened, that's what she should have done.
"When you get to that level, you are subject to all sorts of political attacks, allegations and accusations of not telling the truth. That's part of politics I'm afraid."