Citizens Advice is accusing councils of letting bailiffs get away with threatening and aggressive behaviour when collecting council tax debts.
The charity claims that councils can make the debts worse by adding on charges.
It said there was a huge variation in these costs in different areas of England and Wales.
But the bailiffs' trade body said the Citizens Advice report was based on "distorted facts".
In the year to November, Citizens Advice said that it had heard from 16,905 people worried about bailiffs enforcing council tax debt.
'Pseudo statistical analysis'
The findings of its report was based on a survey of 500 people who had bailiffs chasing them for council tax debts.
Of these, 38% said that they were charged fees for visits bailiffs never made. Some 40% reported that they were threatened with the removal of items that did not belong to them.
However, the Civil Enforcement Association, which represents bailiffs, has hit back at the claims.
"This is based upon distorted facts, the use of pseudo statistical analysis and highly emotive and inappropriate language," said Steven Everson, director general of the association.
"This self-selecting sample of 500 unhappy individuals cannot be extrapolated to imply that it reflects the situation amongst the general population of debtors."
Separately, research by Citizens Advice suggested that a quarter of councils did not hold any data on complaints about bailiffs.
Debts that are collected by bailiffs can include bailiff fees. When councils pass debts to a bailiff they must first obtain a liability order from court, with many authorities charging people for the liability order.
Citizens Advice said that the amount charged ranged from £3 to £126.
Councillor Sharon Taylor, who chairs the Local Government Association's Finance Panel, said: "Councils have a responsibility to their taxpayers to collect taxes so that important services like caring for the elderly, collecting bins and fixing roads are not affected.
"Bailiffs are only ever used by local authorities as a last resort to tackle persistent non-payment and councils only charge fees that cover the costs they incur in processing each liability order.
"Councils that use bailiffs to recover debts have a responsibility to ensure that people are being treated fairly and that resident complaints about unacceptable bullying or intimidation are taken seriously."