Wales' Whitehall department has little influence on the UK government, Welsh First Minister Carwyn Jones has said.
It comes after Welsh Secretary Alun Cairns, who leads the Wales Office, said the EU withdrawal deal was a matter for UK ministers not Mr Jones.
Mr Jones told AMs the effect to international trade and treaties was "huge in devolved areas".
"But I don't think the Wales Office has much influence at all in the way that the UK government operates," he added.
A UK Government source said the Welsh Secretary is a "integral part of government".
Mr Cairns made his comments, on Wednesday, after the Welsh Labour leader said he should have been briefed earlier on the Brexit agreement.
Mr Cairns said: "This is a reserved area of policy. There are two governments in Wales.
"I don't tell the Welsh Government what to do on health policy and this is a matter that is a European policy as well as a UK government policy."
Mr Jones responded on Friday at an assembly committee that scrutinises his actions as first minister.
When AMs raised concerns about the Conservative minister's statement, Mr Jones explained that Welsh ministers' relationship with the UK government was "not channelled through the Wales Office".
"I deal directly with the prime minister - I don't deal via the secretary of state for Wales and then go through him to the prime minister," he said.
On the Welsh secretary's comments, Mr Jones said: "He's not right to say, of course, that it's a reserved matter in practice.
"Yes, of course, international trade and international treaties are a reserved matter but their effect is huge in devolved areas.
"It's never been wise to simply say 'this has nothing to do with you' because it clearly does."
"But I don't think the Wales Office has much influence at all on the way that the UK government operates.
"And certainly the UK government, I think, finds it easier to deal with us."
A UK Government source said: "The First Minister is very much aware that the Office of the Secretary of State for Wales is an integral part of Government and works closely with Whitehall departments, local authorities and his officials to ensure that the UK Government delivers for Wales.
"The scrapping of the Severn tolls, the work to deliver city and growth deals for every part of Wales, and the securing of £550 million from the budget for Wales are a testament to the department which works tirelessly to secure a better future for Wales."