New guidance on the use of anti-social behaviour powers has been issued to ensure they are reasonably applied.
The Home Office told councils and police in England and Wales the laws should not be used on the vulnerable.
It comes amid concerns orders were being used to target buskers, rough sleepers, dog walkers and groups gathering to chat in town centres.
Victoria Atkins, minister for crime, safeguarding and vulnerability, said powers should be used proportionately.
She said: "We know that these powers are being used to very good effect by the police and local councils across England and Wales, and we are very keen to encourage their continued use.
"But we are also clear that the powers should be used proportionately to tackle anti-social behaviour, and not to target specific groups or the most vulnerable in our communities."
The guidance follows feedback from charities and other groups that the orders were being used to disproportionately target certain groups, including rough sleepers.
Hundreds of fines have been issued for violations such as playing music too loudly in cars and not having a dog on a lead.
Martha Spurrier, director of campaign group Liberty, said some councils were "compassionless" in applying orders on vulnerable people.
"In the last year, very sadly, six people have been sent to prison for being homeless as a result of these orders," she said.
"They've gone very off-piste and have been used by these local councils in a pretty compassionless way - so that poverty effectively ends up being criminalised."
Local agencies, including councils, police and social landlords, have six powers to tackle anti-social behaviour - the civil injunction, criminal behaviour order, community protection notice, public spaces protection order, closure power, and the dispersal power.
They were introduced in 2014 after the government overhauled old anti-social behaviour powers, believing they were ineffective.
The revised guidelines state orders should be focused on "specific behaviours and are proportionate to the detrimental effect that the behaviour is causing or can cause, and are necessary to prevent it from continuing, occurring or recurring".
Anyone issuing an order should ask themselves if the behaviour in question is unreasonable, the guidance states.
"For instance, a baby crying in the middle of the night may well have a detrimental effect on immediate neighbours and is likely to be persistent in nature," it said.
"However, it is unlikely to be reasonable to issue the parents with a Community Protection Notice if there is not a great deal that they can do to control or affect the behaviour."
Simon Blackburn, chairman of the Local Government Association's safer and stronger communities board, said: "Many anti-social behaviour offences are serious issues for local residents and businesses, and councils are keen to protect them from offenders who can make the lives of those they target a misery.
"Councils will take a proportionate approach to using the tools at their disposal to tackle crime and anti-social behaviour."