Carers are being bullied by the government, a committee of MPs says.
The Work and Pensions Select Committee says ministers are harassing people who have made honest mistakes when receiving carer's allowance.
Long-standing problems within the Department for Work and Pensions (DWP) led to a failure to spot overpayments.
The government says while it has a duty to the taxpayer to recover the money, "safeguards are in place to protect claimants from financial hardship".
The MPs have urged the government to review, on a case by case basis, whether overpayments are worth pursuing, given the Department of Work and Pensions' culpability and the cost of recouping them.
"Bullying carers is no way to recognise, much less support, the invaluable contribution they make to our society and the people they care for," said Frank Field, who chairs the committee.
"The DWP and the courts are not listening," George Henderson, from Leyland, Lancashire, says.
He is being pursued for an overpayment of more than £19,000 of carer's allowance by the DWP - and has had to put his house up for sale.
The 59-year-old first started receiving the benefit in 2010 while caring for his son John, who has mental health problems and is a heroin user.
He applied for carer's allowance and stopped his son receiving an alternative benefit, as John's drug dealer "actually had his bank card".
"He knew when he got paid, he went and got his money out of the machine, gave him the drugs and that was it," George says.
George admits he made a "mistake" when he applied, stating he was unemployed when he was actually working for a taxi company.
In 2017, the DWP contacted him to say he was not eligible for the benefit.
He was convicted of fraud, given a 32-week suspended sentence and electronically tagged for 16 weeks.
The DWP then used a proceeds of crime order to make George to sell his home.
Over seven years, George is estimated to have received about £100 more than his son would have been paid if he had continued to receive the benefit he was on.
"The impact has been immense," he says.
"There was no money to be made.
"If I'd known that I was earning too much money, obviously I'm not going to claim carer's allowance, take money off my son to give it back to him on a daily basis, and then get myself done for fraud.
"I mean it's just ludicrous. It's laughable."
For years, the Department for Work and Pensions ignored a whistleblower's concerns they were failing to spot overpayments to thousands of carers as they had cut the number of staff working on the benefit.
HM Revenue & Customs was providing earnings data to the department that should have highlighted the overpayments, while the confusing nature of the benefit meant many people were unaware they were no longer eligible.
In April, the National Audit Office said about two-thirds of carers with debts for earnings-related overpayments over £2,500 would have had their overpayments stopped earlier if the DWP had put in place sufficient staff.
Carer's benefit is paid to people who provide at least 35 hours a week of care.
The government says 850,000 people receive the allowance, with millions of payments made each year.
It is currently worth £66.15 per week and the recipient is allowed to earn up to £123 per week and still receive carer's allowance, though their earnings can be higher if they have childcare costs - but the moment they earn a penny more, they immediately become ineligible and have to repay any benefit they may have received.
The National Audit Office found the DWP was aiming to recoup overpayments from 80,000 people, worth about £150m.
Most debts are for less than £1,000 but some people owe more than £20,000.
BBC News has discovered the DWP and the Crown Prosecution Service, who take the cases to court, are not routinely telling judges the department could have spotted the errors earlier.
A Crown Prosecution Service spokesman said every case was "carefully considered" but it did not have a duty to disclose "publicly available government reports that do not relate to specific cases".
Welfare experts told BBC News, however, it was unrealistic to expect judges or magistrates to be aware of the changing circumstances around any individual benefit.
Emily Holzhausen, director of policy and public affairs at Carers UK, said many people who have been impacted by overpayments were already struggling financially and are experiencing considerable stress and anxiety.
"The department must urgently consider writing off overpayments where its administrative failures have allowed them to accrue," she added.
Shadow work and pensions secretary Margaret Greenwood said carers are "paying the price for a rigid system that is poorly explained".
The DWP said decisions to prosecute were not taken lightly and were generally reserved for overpayments of more than £5,000.
"It is a condition of receiving carer's allowance that people tell DWP if their circumstances change and we work extremely hard to make claimants aware of their responsibility to provide correct information," a spokeswoman said.