It is an "absolute priority" for the government to leave the EU by 23 May to avoid having to take part in European elections, Jeremy Hunt has said.
The foreign secretary said the public would find it "hugely disappointing" to be asked to send MEPs to Brussels.
Asked if it could be a disaster for the Tories, he told the BBC "in terms of polling it certainly looks that way".
Some local Tory activists have signalled they will not campaign and regard the polls as a "distraction".
Downing Street said that in order to avoid the need for elections, legislation implementing the Brexit withdrawal deal would have to be passed by Parliament by 22 May.
Last week, the EU agreed a new Brexit deadline of 31 October.
Talks between the government and Labour are set to continue over the Easter parliamentary recess in the hope of finding an agreement that will be acceptable to MPs.
A series of working groups in key areas, such as environmental standards, security and workers' rights, have been set up to try and find common ground.
Speaking on a visit to Japan, Mr Hunt said the talks with Labour had been "more constructive than people thought" but "we don't know if they are going to work".
If they did not lead anywhere, he suggested the government may "need to find a way to rebuild the DUP-Conservative coalition", which has come under real strain from Brexit.
The Democratic Unionists are supposed to support the government in key parliamentary votes to give it a majority in the House of Commons.
But they have refused to support the prime minister's withdrawal agreement over concerns with the controversial Irish backstop, which aims to avoid a hard border on the island of Ireland.
Mr Hunt told BBC Radio 4's Today programme that continuing Brexit "paralysis" would be "highly damaging" to the UK's global standing and international trading partners "are worried that we will become submerged in the mire of Brexit indecision".
While Japan and other major foreign investors were keen for the UK to "make up its mind" about Brexit, he suggested they would continue to keep faith with the UK even if it left without a deal.
"(Japan) has signed a deal with the EU and, in a no-deal situation, we hope that would roll over and apply for us, although no-deal, I think, is looking much less likely," he said.
"I think they are very keen to protect their trading relationship with the UK, but I think they are also wanting to talk to us about other things."
However, in February, the government released a statement saying it would not be able to replicate the EU's free trade deal with Japan after Brexit.
And Japan has not agreed to continuing existing trade terms in the event of no deal.
Mr Hunt downplayed talk of an imminent Conservative leadership contest, saying it would be a "sidetrack" from Brexit.
The foreign secretary, along with his predecessor Boris Johnson, are among a long list of potential candidates touted to succeed Mrs May when she stands down.
Asked whether the next leader could be someone, like himself, who campaigned to remain in the EU, Mr Hunt replied: "There is one very big difference between me and Boris, which is that I am foreign secretary and I have a very big job to do to try and get this deal over the line and that has to be my focus.
"I think that what matters is we have a cabinet that believes in Brexit."
On the first day of his trip to Japan, Mr Hunt met the country's prime minister Shinzo Abe and other political leaders.
He also took some time out of his political schedule to talk to pupils in a school in Tokyo. Mr Hunt is a fluent Japanese speaker, having taught English as a foreign language in Japan in his 20s.
If my kids grow up to be half as intelligent and well behaved as the children at Tokyo Metropolitan Hibiya High school this morning I’ll be a very proud Dad. Great to be teaching English in Japan again after a twenty year gap. pic.twitter.com/72wUkfRI8P— Jeremy Hunt (@Jeremy_Hunt) April 15, 2019