Debate about Britain's imperial past and the Black Lives Matter movement has a "cartoon-like view" of history, Kwasi Kwarteng has told the BBC.
The business secretary, the first black MP to head a department, told Nick Robinson's Political Thinking the Empire must be seen in its 400-year context and arguments fully explored.
Calls to "de-colonise" the curriculum were not the way forward, he added.
"I'm saying the opposite - that you've got to learn more about colonialism."
British institutions have faced growing calls in recent years to account for their colonial links and wealth they accrued directly as a result of the British Empire and the Atlantic Slave trade.
During anti-racism protests last summer following the death of George Floyd in the US, the statue of slave trader and philanthropist Edward Colston was toppled by protesters in Bristol and a number of others were removed.
Mr Kwarteng, whose parents moved from Ghana to the UK in the 1960s, said "ripping down" statues was counter-productive, as well as illegal, and people should "learn more" about the British Empire in all its various facets.
"I'm not someone who is obsessed with trying to correct history. I think you've got to understand history and understand that it's very complex and there are different arguments that are presented and I have a plea to understand the arguments and the context of the British Empire," he said.
"Even when you say the phrase the British Empire, you're talking about something that lasted more or less 400 years and covered a huge expanse of territory.
"So within that time and geography there's a huge amount of variety, different cultures and different time periods and getting a sensitivity to that is hugely important.
"I think a lot of the debate around Black Lives Matter and imperialism or colonialism has a very kind of cartoon-like view of what was happening over centuries across a quarter of the world."
He said he did not understand what those calling for university and school curriculums to be "de-colonised" were trying to achieve.
"Is the implication that it's a colonial relic and that you've got to try and decolonise it? I'm saying the opposite - that you've got to learn more about colonialism."
During the interview, Mr Kwarteng also explained why he put a stop to a post-Brexit review of workers' rights and employment practices when he took over at the business department three weeks ago.
"I don't think that's a constructive thing to do. I don't think the opportunities of Brexit are about whittling away workers' rights or trying to have a race to the bottom or trying to reduce wages. On the contrary, a lot of people in my constituency voted for Brexit because of a stagnation of wages."
Mr Kwarteng was one of the authors of a 2012 series of essays, called Britannia Unchained, which suggested the UK did not encourage risk-taking and must strive to emulate the work ethic and low-tax culture of Singapore and other Asian successful economies.
Pressed on whether he still holds to pro-free market views on deregulation or whether he supported a more interventionist role for the state, Mr Kwarteng said 2021 was very different from 2012, let alone previous decades.
"We've had a huge amount of money spent rightly to support businesses and people in employment. We've got a net zero agenda, which says we've got to decarbonise our economy in 30 years, and we've got a Brexit deal which is going to set us on a totally different course in terms of international trade.
All of these mean that we can't simply apply the orthodoxies of 30 or 40 years ago to the context in which we now live or even five or six years ago.'
He added: "You can't simply go back to the Milton Friedman and Margaret Thatcher playbook of the 1970s and 1980s of deregulation and apply that.'
The full podcast will be published today and Political Thinking will also be on Radio 4 tomorrow (Saturday) at 5.30pm.