It is not possible to see how the Irish border issue can be resolved after Brexit, the influential group of MPs scrutinising the process has said.
The government wants no hard border between the Irish Republic and Northern Ireland and no customs border between the latter and the rest of the UK.
Ministers have suggested technology could enable a "frictionless border".
But the Committee for Exiting the EU said the proposals were "untested" and "to some extent speculative".
Ireland's foreign minister Simon Coveney told the BBC his country "can't be asked to leap into the dark" by simply accepting UK assurances.
Northern Ireland will be the only part of the UK to share a land border with an EU member state after the UK leaves.
There is currently no physical infrastructure on the border but there is concern that this will have to change after Brexit.
If the UK leaves the EU's single market and customs union, as the government intends, the Irish land border will become the external border for the EU's single market and customs union.
The Irish Republic wants Northern Ireland to keep following EU rules, so that goods can continue moving across the border - in effect, staying within the customs union and single market.
But this would effectively push the customs border out into the Irish Sea, becoming an internal customs border between Northern Ireland and the rest of the UK - which the UK government rejects.
In its report, the Exiting the European Union Committee says it does not see how it will be possible to reconcile these positions.
The government has put forward two proposals, one using "technology-based solutions", such as pre-screening of goods and trusted trader schemes, to reduce the need for customs checks at the Irish border.
The other would involve a "customs partnership", with the UK leaving the single market without introducing an EU-UK border - something the UK has admitted would be "challenging".
The committee is urging the government to set these proposals out in more detail.
"Ministers say they don't want a border, they don't want any infrastructure," said its Labour chair Hilary Benn.
"We all agree with that...but self-evidently, the Irish government is not persuaded by what it has read so far."
UK Prime Minister Theresa May has said the UK and Irish governments "have the same desire" on the border - to ensure that the movement of trade and people continues "as now" and that no new barriers are created.
The government added that it remained "absolutely committed to finding a solution that works for the people of Northern Ireland and Ireland".
But Irish foreign minister Simon Coveney said it would be hard to avoid physical checks if there were different standards for animal welfare, food safety and medical regulation on either side of the border.
"What the British government has been asking of the Irish government is 'just trust us we'll solve these issues with a broad bold trade agreement'," he told Radio 4's Today.
"But that may not be possible, we don't know. We can't be asked to leap into the dark by opening up phase two discussions in the hope that these issues might be resolved."
He added: "The area that we've focused in on is the need to give reassurance that there will not be regulatory divergence between the two jurisdictions on the island of Ireland
"Because if there is it is very hard to avoid a checking system."
Northern Ireland's Democratic Unionist Party has warned that any attempt to "placate Dublin and the EU" could lead to the end of its confidence-and-supply agreement with the UK's Conservative government.
The DUP struck a deal with the minority Conservative government in June, agreeing to support Tory policies at Westminster, in return for an extra £1bn in government spending for Northern Ireland.
The Committee for Exiting the EU itself was split over the report, with five of its 21 members - four Conservatives and a DUP MP - voting to reject it.
The report also includes a call for the government to publish the likely terms of any transition period governing what will happen immediately after Brexit in March 2019.
It says it is "essential" that the details of the arrangements be published by the end of March so as to give businesses enough time to prepare.