Boris Johnson has rejected a demand from the first minister of Wales for a travel ban in and out of Covid hotspots in England.
"There are no physical borders between Wales and England," the prime minister's official spokesman said.
On Monday Mark Drakeford threatened to take action if a ban is not imposed.
The UK government has given advice for people in Merseyside not to travel in or out of the area - but that has stopped short of a legal ban.
The prime minister's spokesman said the guidance was "very clear".
Plaid Cymru called for the first minister to act, but the Welsh Conservatives accused Mr Drakeford of relying on evidence drawn up on the "back of a fag packet".
Meanwhile Mark Drakeford said a special Cobra meeting should be held to discuss a so-called "circuit-breaker" lockdown.
Welsh ministers have asked for travel from areas with high rates of coronavirus in England to be restricted, to prevent people visiting parts of Wales where lockdowns are not in force and where infection rates are lower.
It is the second time the UK government has said it would not agree to the Welsh Government's request.
Seventeen places in Wales - 15 counties, one town and one city - are currently under local lockdown restrictions, effectively banning non-essential travel.
Mr Drakeford wrote to Mr Johnson on Tuesday saying efforts in Wales "are being undermined by travellers from high-prevalence areas in other parts of the UK travelling to Wales".
He attached a scientific paper which he said "demonstrates the spread of infection geographically and supports the case for travel restrictions as a means of controlling the spread of the virus".
The Welsh Labour leader wrote to the first ministers in Scotland and Northern Ireland and asking them to regulate travel in a similar way.
"It would be better if all four nations were to act in concert, but in the absence of an agreed way forward, I will act to keep Wales safe," he said.
The Scottish Government said it understood Mr Drakeford's concerns and First Minister Nicola Sturgeon would respond "in due course".
The paper, cited by Mr Drakeford and seen by BBC Wales, concludes that the "easing of lockdown rules into August has corresponded with an increase in cases [in Wales], which may partly be driven by imports from other parts of the UK and wider world".
It adds that cases "in less urban areas" are "more likely to be imports from elsewhere and rarely lead to local onward transmission".
But the paper also said the data "does not constitute definitive proof".
UK government sources were told the paper had not been peer-reviewed enough to be published.
But Welsh Health Minister Vaughan Gething said in the Senedd on Tuesday that the time it would take for a peer-reviewed paper "would mean that we would have to wait a very long period of time before we could take any action".
What do people on the border between England and Wales think?
Tanat Holiday Park in Llanymynech sits on the border - with 90 caravans in England and 45 in Wales.
Owner Cheryl Hall said she had found it challenging keeping customers up to date on ever-changing regulations and who can and cannot visit.
"There is advice on non-essential travel," she said of the advice in Merseyside, which will go into the very high tier of lockdown on Wednesday.
"But it's only guidance, not law. It's not up to me to police it."
She said people from south Wales know they cannot come to the park.
"I had a customer who was here for most of the summer but when she left to go home, she knew she wouldn't come back. We don't know when we will see her again."
Janette Harrop, from St Helens, in Liverpool, lives in a caravan at the park. She said: "We haven't been home for over three months. We feel safer in the caravan.
"We are hoping they stop people coming into Wales, and those of us who are already here and have been here a while can stay."
She said: "Where we are from in St Helens, our area Castle Heath is quite high as well, so we are better off here where it's low, than going home and risking catching corona."
Analysis by Ione Wells, BBC Wales parliamentary correspondent
What UK ministers want to know is whether it can be categorically proven that people travelling from hotspot areas in other parts of the UK has led to an increase in Covid-19 cases in Wales.
It's unclear whether evidence of cases being imported, without saying exactly where from or who by, will cut it.
Without this evidence, UK government sources say it is pretty unlikely ministers will budge and put this current travel guidance into law.
They also have questions about how laws would be enforced - something Welsh ministers insist is possible.
And without an answer from the PM or FMs of Scotland and Northern Ireland, Welsh ministers say they will have to introduce their own measures.
'Time to act'
Plaid Cymru leader Adam Price said the UK government "has shown utter contempt for Wales, our people and our democracy.
"The time to ask is over. Now is the time to act."
Welsh Conservative health spokesman Andrew RT Davies said it was "deeply embarrassing" for Mr Drakeford and Labour.
"He and his colleagues should hang their heads in shame, because pursuing such a controversial policy on back-of-a-fag-packet evidence is no way to run a government at any time, least of all during a pandemic and lockdowns that are crippling businesses and lives."
It is not clear what the Welsh Government will do next.
BBC Wales was told there will be discussions across cabinet, with no timetable for an announcement.
The first minister said in the Senedd he has asked the prime minister for a special Cobra meeting to discuss a "circuit breaker" system - imposing a short period of restrictions for everyone - for dealing with coronavirus.
The UK government's Sage advice group called for a short lockdown three weeks ago.
Mr Drakeford said the Welsh Government's Technical Advisory Cell (TAC) had not yet advised ministers to implement such a system in Wales "but I do take the arguments in favour of a circuit breaker period seriously".
"I think it's an idea that will need further examination and needs to be shared in perspective between the four UK nations," he said, adding the request was in Tuesday's letter.
Mr Drakeford added that his call for a travel ban was not "some sort of contest between Wales and England".
"This is not about stopping people from England coming to Wales, nor should we ever fall into that sort of way of talking," he said.
The prime minister's spokesman said: "What we have done is publish guidance which is very clear that people from very high risk areas such as Merseyside should avoid travelling in or out of the area.
"We have also made it very clear to the public that they should follow any local guidance which is issued by devolved administrations."