Nearly six million people in the UK have paid the wrong amount of tax.
About £2bn was underpaid via the Pay as You Earn (PAYE) system in the past two years, with about 1.4 million people owing an average of £1,500 each.
But £1.8bn has also been overpaid and some 4.3 million people will get a rebate because they have paid too much.
Treasury minister David Gauke said that in the current financial climate, the government was not in a position to "just wave goodbye" to the money owed.
He said the government had inherited the problem and the PAYE system - which was created in the 1940s - was struggling to cope with modern working patterns.
A spokesman for HM Revenue & Customs (HMRC) rejected suggestions that as many as 10 million people might be eligible for a rebate because there is a backlog of unresolved cases from the years before 2008-09.
"We don't recognise the 10 million figure, just because a case is open does not mean a refund is due," he said.
A new computer system introduced by HMRC in 2009 has allowed more discrepancies to be identified.
As a result millions of letters will be sent to taxpayers across the UK informing them of errors in their contributions.
The first 45,000 are expected to arrive on Tuesday, with 30,000 informing recipients they are due a rebate of on average £418.
The remaining 15,000 letters will tell taxpayers they have underpaid and will have their tax code altered next year to recoup the money.
It is thought that some individuals may face both underpayments and overpayments, which could cancel each another out.
Discrepancies arise when the amounts deducted in tax and National Insurance by employers using the PAYE system do not match the information held on HMRC records.
This most often occurs when individuals change jobs, have more than one job at the same time, or because employers are using the wrong tax code.
In some cases officials say they will consider writing off demands for additional money if taxpayers can demonstrate they provided all the information necessary to calculate their tax correctly.
An HMRC spokesman told the BBC: "The overwhelming majority of PAYE cases - over 40 million - are right, so most people have paid the right amount of tax.
"But for a variety of reasons in some cases there will be a discrepancy.
"The government accepts that the way we go about deducting tax at source needs to be much more accurate and the introduction of the NPS [computer system] paves the way for a real time system which in turn boosts accuracy."
John Whiting, from the Chartered Institute of Taxation, told the BBC that some of the poorest, including those who received means-tested benefits, could have been hit twice, as their benefits would have been incorrectly calculated.
"It is very difficult to go back and claim benefits you underclaimed, whereas, as demonstrated, if you owe tax it is possible for the revenue to back claim there," he said.
Mr Gauke said the government wanted to move sensitively and cautiously but he recognised the difficult situation some people were facing was "through no fault of their own".
He added: "At the moment we have said that those who owe more than £2,000 - those who are obviously in the most difficult position - we're reviewing exactly how we're going to do that.
"For those who owe less than that we will be seeking to recover that over the course of the 2011-12 tax year through tax codes."
George Mudie, a Labour MP and member of the Treasury Select Committee, told the BBC he believed there was a case for waiving the debt owed by those who had underpaid.
He said he believed the HMRC had handled the situation badly and unfairly.
He described the HMRC as a "heartless" and "arrogant" organisation and said: "If they want money, they take the money and very rarely are they prepared to consider, even when they make mistakes, not taking the money."
Emma Boon, from campaign group the Taxpayers' Alliance, said the HMRC must take steps to help those told they have to pay more.
"Some of them won't be finding out about it for a few weeks or maybe even a few months, so it could be towards Christmas which really isn't what you want to hear," she said.
In June, the government ordered a review of how the PAYE system works and is encouraging the public to contribute their thoughts about how it could be improved.