Google's £130m UK tax settlement was "not a glorious moment" for the government, Business Secretary Sajid Javid has acknowledged.
The UK tax authorities' agreement with Google has fuelled a "sense of injustice" that big businesses receive preferential treatment, he said.
His comments come a week after Chancellor George Osborne called the settlement a "major success".
Google said it was in favour of reforms to make international tax clearer.
However, shadow chancellor John McDonnell warned the fallout from the tax row had damaged Google's brand.
"The reputational damage to Google, I think, is immense," he said, and suggested the "saving they have made in tax is not worth the reputation damage they have had".
Mr McDonnell also repeated a call for Google and HMRC to release details of how the agreement was struck.
He has published his tax return, showing he paid £14,253 in tax last year, and urged the chancellor to do the same, saying taxpayers filling in their own tax returns, which are due by midnight, would be "feeling angry" about the Google settlement.
Speaking on the BBC's Andrew Marr Show, Mr Javid agreed that "more needs to be done" to ensure multinational companies pay the correct corporation tax.
"I speak with thousands of companies small, medium-sized as well as, of course, large companies and there is a sense of injustice in what they see," he said.
But the Google agreement could be seen as a "success" in so far as it would help to change the behaviour of other large companies, he said.
Big businesses will see that HMRC "will not give up, they will come after you if they feel you're not paying your fair share in taxes," he said.
Earlier, a senior Google executive said the company felt the global tax system, which dates back to the 1920s, needed to be updated.
Peter Barron, a senior Google communications director, told the Marr Show: "We would like to see more simplicity and more clarity, not least because we would like to be seen to be paying the right amount."
Google still follows its motto of "Don't be evil" and tries to do the right thing, including on tax affairs, he said.
UK sales account for around 10% of Google's global revenues, but the tech giant says UK profits, which is how corporation tax is calculated, are lower.