Downing Street says it is "unlikely" the UK's Rwanda asylum policy will come up when Boris Johnson meets the Prince of Wales on Friday.
The prime minister had said he would defend the scheme if it was raised, but his spokesman later said it would not be at the forefront of his mind.
Prince Charles is reported to have criticised plans to remove asylum seekers to the African country.
The pair are in the Rwandan capital Kigali for a Commonwealth summit.
The prime minister's spokesman said the policy was not something the Mr Johnson was focused on at the Commonwealth Heads of Government Meeting (Chogm) summit, and emphasised challenges facing the Commonwealth such as sustainability, climate change and girls' education were the priority.
A Clarence House spokesman said he would not be commenting on "supposed remarks made in private" except to say that "the prince is politically neutral" and that policy was a matter for the government.
The policy has been criticised by politicians, refugee organisations and the Church of England, with some pointing to Rwanda's human rights record.
There have been several legal challenges in the UK against the policy, with the first flight due to take people to Africa prevented from taking off earlier this month.
Earlier, the prime minister said Rwanda had "undergone an absolute transformation" in the last couple of decades and had come on "leaps and bounds" in education and "taking society forward".
"People need to keep an open mind about the policy, the critics need to keep an open mind about the policy," he said.
It was reported in the Times, that Prince Charles had privately criticised the plan of sending asylum seekers to Rwanda as "appalling".
But the prime minister, who said he had discussed the matter with Rwandan President Paul Kagame, said it was "absolutely necessary and right to fix the problem of illegal cross-Channel trafficking of people".
Prince Charles, who is with his wife the Duchess of Cornwall, has met business leaders in Rwanda as part of the Commonwealth Business Forum.
Under the Rwanda asylum plan, announced in April, the government intends to remove some asylum seekers who cross the Channel to the UK, sending them via a one-way ticket to Rwanda to claim asylum there instead.
It has been criticised with some pointing out that last year the UK had raised concerns about restrictions to civil and political rights in the country.
After the first flight taking asylum seekers to Rwanda was cancelled, Labour's shadow home secretary Yvette Cooper said the scheme had "never been a serious policy" and described it as a "shambles" and "shameful".
When the policy was first announced, the Liberal Democrats accused the government of "slamming the door" in the face of refugees.
During his visit to Rwanda, Mr Johnson will not be visiting any of the accommodation set aside for the scheme, with his spokesman saying his time was limited.
"We think that the best use of his time for this short period he's in Rwanda is to dedicate himself to some of the issues that will be raised at the summit and to work with other world leaders on some of those issues we've talked about, not least Ukraine and global security," he said.
During a speech as part of the summit, Mr Johnson spoke of the UK's support for green energy projects in Africa, including the announcement of up to £162m investment, via British International Investment, in the hydropower sector.