HMRC has rejected criticism by Andrew Tyrie, chairman of the Treasury select committee, over its response to the exploitation of tax breaks for the film industry.
The Tory MP has written to Chancellor Philip Hammond, to express a "number of concerns".
The tax office has cracked down on a number of schemes that created artificial losses for investors.
Those losses allowed investors to lower their overall tax bills.
Mr Tyrie said in his letter that his office had been contacted by an increasing number of people concerned that HMRC investigations into such schemes were "not always fair nor what anyone could have expected".
That had resulted in "financial calamity" for some individuals, as well as considerable difficulties for HMRC in closing down some schemes.
"Many have said that, when these schemes were being sold, they were not considered to be aggressive avoidance but just a deferral of tax, and they were often marketed as routine tax management," Mr Tyrie told the Chancellor.
"Whether or not these claims are valid, it does appear that many individuals are facing very severe financial distress as a consequence."
An HMRC spokesperson said: "We have worked hard to tackle abuse in the system on behalf of the vast majority of investors who play by the rules, ensuring they are enforced fairly, and with sensitivity."
Commenting on the correspondence, Mr Tyrie said: "If a tax avoidance scheme is found by the courts to be illegal, then investors in the scheme have no option but to repay the tax that they have avoided.
"But HMRC needs to treat people fairly, including investors in what are now considered to be tax avoidance schemes, and to be transparent and timely in their approach."
Some investors in film schemes have been hit with Accelerated Payment Notices, which give taxpayers 90 days to pay the disputed amount of tax.
Their use has been controversial because some tax advisers claim they scare taxpayers into paying up rather than contest claims in the courts.
Some 60,000 notices have been issued since their introduction in 2014 and have raised £3bn.
Although some consider the notices to be draconian, as they can require taxpayers to stump up sums described as "life-changing", HMRC has won various attempts to challenge their legality.
In August last year HMRC claimed victory in a tax avoidance battle over schemes worth more than £820m run by Ingenious Film Partnership and Icebreaker.
The Ingenious scheme tried to use artificial losses arising from backing a range of films including Avatar, Life of Pi and Die Hard 4. while the Icebreaker scheme attempted to create artificial losses from limited liability partnerships.
HMRC said both schemes saw users claim more in tax relief than they had invested - a claim that Ingenious disputed.
Ingenious called the Tax Tribunal decision "arbitrary and subjective", and a spokesman for the investment company said HMRC had deliberately misrepresented the judge's findings in its own case by conflating it with that of Icebreaker.
"The question in Ingenious's case therefore is not whether these were vehicles set up purely for avoiding tax (as they were found to be in Icebreaker) - they were found to be bona fide businesses," he said.
"The problem is on the level of tax relief investors are able to claim which the judge found to be 30% not the 100% originally envisaged by Ingenious.
The accounting computations relating to this judgment are subject to continuing representations by both parties," the spokesman added.