There can be no final decisions on the future of the Irish border until the UK and the EU have reached a trade agreement, Liam Fox has said.
The UK's international trade secretary also blamed the EU for Brexit delays.
The comments came after the Irish Republic's EU commissioner said Dublin could veto Brexit trade talks.
The EU has said "sufficient progress" has to be made on the Irish border before negotiations on a future relationship can begin.
Downing Street has said the whole of the UK will leave both the customs union and the single market when it leaves the EU in 2019.
"We don't want there to be a hard border but the UK is going to be leaving the customs union and the single market," Mr Fox told Sky News.
He added: "We can't come to a final answer to the Irish question until we get an idea of the end state. And until we get into discussions with the EU on the end state that will be very difficult - so the quicker we can do that the better, and we are still in a position where the EU doesn't want to do that."
Mr Fox accused the European Commission of having an "obsession" with ever-closer union between EU member states, which was delaying progress in Brexit talks.
Phil Hogan, the EU's agriculture commissioner, told the Observer that staying in the customs union would negate the need for a hard border - with customs posts and possible passport checks - on the island.
He said Dublin would "play tough to the end" over its threat to veto trade talks until it had guarantees over the border.
Shadow chancellor John McDonnell said he was "worried" by Mr Fox's comments, adding that Labour would not take continued membership of the single market and the customs union off the table.
"I think the one thing that we don't want to do is jeopardise any movement quickly, because we need movement to enable us to get into the proper trade negotiations," Mr McDonnell told ITV's Peston on Sunday.
"So I'm hoping that isn't a Downing Street-sanctioned statement that's he's made."
By Chris Mason, BBC political correspondent
It's 310 miles (499km) long - a squiggle on the map that meanders from Carlingford Lough in the east to Lough Foyle in the west.
The border between Northern Ireland and the Republic is the soon-to-be frontier between the UK and the European Union.
And right now it is the most troublesome frontier between Brexit negotiations stalling or progressing.
London and Dublin each say they are committed to maintaining an open border. But Ireland wonders how that will be possible.
Oh and one other thing to throw into the mix - after all the talk of how wobbly Theresa May's government is, so is Ireland's.
There could be a general election there before Christmas.
The EU has given Prime Minister Theresa May until 4 December to come up with further proposals on issues including the border, the Brexit divorce bill and citizens' rights, if European leaders are to agree to moving on to trade talks.
But Mr Hogan accused some in the British government of having what he called "blind faith" about securing a comprehensive free-trade deal after Brexit.
He said it was a "very simple fact" that "if the UK or Northern Ireland remained in the EU customs union, or better still the single market, there would be no border issue".
In these circumstances regulations on either side of the border would remain the same, and so a near-invisible border would be possible.
The Irish government has always insisted there must not be a hard border between the Republic and Northern Ireland, with Taoiseach Leo Varadkar saying he must have written assurance from the UK before Brexit talks can move on.
Irish Foreign Minister Simon Coveney has said the UK's desire for no hard border on the island of Ireland was "aspirational".
It comes as Ireland's deputy prime minister faces a motion of no confidence over her handling of a case involving a whistle-blower alleging corruption within the police.
The issue could see Prime Minister (Taoiseach) Leo Varadkar's coalition government fall and an election held before Christmas.
In her speech in Florence, this September, Mrs May restated that both the UK and EU would not accept any physical infrastructure at the border.
The Democratic Unionist Party said Northern Ireland and the rest of the UK must not be different.
Arlene Foster, the leader of the DUP, which is in a confidence-and-supply arrangement with the Conservative government, said she would not support "any suggestion that Northern Ireland, unlike the rest of the UK, will have to mirror European regulations".
Suggestions for alternate arrangements have included a new partnership that would "align" customs approaches between the UK and the EU, resulting in "no customs border at all between the UK and Ireland".