Former Cabinet minister Iain Duncan Smith has been accused of not understanding the role of trade envoys after he called for the removal of those who are against Brexit.
There are 20 MPs and peers from different parties who act as trade representatives abroad.
Mr Duncan Smith said it was "absurd" that this list should include "people who are viscerally opposed to Brexit".
Trade envoys have responded that Brexit is irrelevant to their work.
The former work and pensions secretary was commenting on the fact that some of the current trade envoys voted against the government during the passage of the bill which enabled the triggering of Article 50.
He told the website BrexitCentral: "It is quite unacceptable and high time we appointed new people who believe that what we are doing is positive and full of new opportunity."
But one of the envoys in question - Lib Dem Baroness Northover, who represents the UK in Angola, told the BBC: "He seems to have misunderstood the role of the UK's trade envoys."
She added: "We worked to increase trade before the referendum, and we continue to do so now... Where we stand on the referendum is irrelevant."
The envoy to Oman, Conservative former defence minister Lord Astor of Hever, said he and his colleagues were "doing a huge amount of work around the world, but we'd be doing this whether or not we are in the EU".
He specified that Brexit "hasn't been mentioned" in his interaction with Oman.
He stressed that, like Iain Duncan Smith, he wants to "follow the will of the people in leaving the EU".
A government spokesman said: "The trade envoy programme is a cross-party network of parliamentarians from both Houses whose role is promoting trade and investment between the UK and overseas markets.
"The voluntary roles are appointed by the prime minister and all envoys bring a wealth of experience to help support British businesses in growing and succeeding internationally."
The programme was set up in 2012 under David Cameron to help the UK build links with markets where there is less ministerial engagement, usually developing countries.
The BBC understands all the envoys recently received a letter from the international trade secretary, Liam Fox, confirming they would continue in their roles and the government sees their work as "crucial".