Dominic Raab says he has "full respect" for Black Lives Matter campaigners after he was criticised for comments about "taking the knee".
The foreign secretary said the gesture "seems to be taken" from TV drama Game of Thrones.
He also said it "feels to me like a symbol of subjugation, subordination, rather than one of liberation".
Labour leader Sir Keir Starmer said Mr Raab was "probably already living to regret" his words.
"It wasn't a very wise thing to say," he added in an interview with ITV Granada.
And Labour's shadow justice secretary David Lammy described his comments as "insulting" and "deeply embarrassing".
Taking the knee is a sign of support for the Black Lives Matter movement.
Speaking to Julia Hartley-Brewer on talkRadio, Mr Raab said he understood the "sense of frustration and restlessness which is driving the Black Lives Matter movement".
He said the act didn't feel like a symbol of "emancipation" but added "I understand people feel differently about it so it's a matter of personal choice."
Asked if he would take the knee himself Mr Raab replied he would only take the knee for two people, "the Queen and the Mrs when I asked her to marry me".
Labour's Diane Abbott criticised Mr Raab tweeting: "Taking the knee began in 2016 with American athletes refusing to stand for US national anthem.
"They were protesting police brutality and racism. But Dominic Raab thinks it comes from Game of Thrones!"
The phrase "bend the knee" features heavily in the popular HBO/Sky series Games of Thrones, in which several factions battle to take control of the Iron Throne and rule over Westeros.
When characters "bend the knee" to heads of these factions they submit to their rule.
Acting leader of the Lib Dems Ed Davey called on Mr Raab to make a "fulsome apology" and suggested he should take the knee himself.
Why are people 'taking the knee' at protests?
NFL quarterback Colin Kaepernick first sat on the bench during the national anthem to protest police brutality and racism - but this developed into kneeling in September 2016, generating national media attention.
Mr Kaepernick said at the time: "I am not going to stand up to show pride in a flag for a country that oppresses black people and people of colour."
Mr Kaepernick's former teammate, Eric Reid, who began the kneeling gesture with him has said: "We chose to kneel because it's a respectful gesture. I remember thinking our posture was like a flag flown at half-mast to mark a tragedy."
Mr Kaepernick and Mr Reid were later joined by a number of other players, before the NFL introduced a rule - later repealed - that teams would be fined if players refused to stand for the anthem.
The gesture has since spread to other sports and protests around the world and has been frequently adopted during protests following the death of George Floyd. Premier League players and staff knelt at the start of matches last night to show their support for the Black Lives Matter protests.
Earlier this month, Sir Keir Starmer was photographed taking the knee with his deputy Angela Rayner.
Mr Raab later sought to clarify his remarks tweeting: "To be clear: I have full respect for the Black Lives Matter movement, and the issues driving them.
"If people wish to take a knee, that's their choice and I respect it. We all need to come together to tackle any discrimination and social injustice."
In addition to being foreign secretary Mr Raab is also the de facto deputy, and stood in for Boris Johnson when the prime minister was ill with coronavirus.
A Downing Street spokesperson said Mr Raab was giving a "personal opinion" when he made the comments.