Gypsies and Travellers have said they are concerned moves to turn trespass from a civil offence to a criminal one could amount to discrimination.
The government wants to increase police powers to force people to move on from unauthorised encampments, which it has said cause misery for local residents.
Billy Welch, the Shera Rom, or Head Gypsy, from Darlington, said this would "criminalise a way of life".
Campaigners are also worried it could threaten the right to roam.
At the moment, trespass is a civil offence, which has meant landowners or local councils usually had to get a court order to evict people who set up camps on private or public land. But if the law was changed police would be able to order people to move to local authority sites.
Mr Welch said he believed this would "disproportionately" affect the Gypsy and Traveller community.
He lives at a permanent site near Darlington, but said there were not enough similar places to meet the needs of his community.
"People have got babies, they've got children, they've got elderly people, they've got sick people", he said.
"The times when we used to camp on the village greens and stay in a farmer's field or on the sides of the road, and we'd be left alone for a few weeks to move on again, are finished.
"It should be compulsory for every borough to provide adequate pitches for the Gypsies they have in the area, as well as some that may be coming and going in the meantime."
The police have also pointed to a shortage of authorised sites.
In response to a previous consultation on the issue in 2018, the National Police Chief's Council and the Association of Police and Crime Commissioners said they were not in favour of changing the law, because "the solution lay in addressing the shortage of suitable sites".
Walking charity the Ramblers has been concerned about the implication for those who may inadvertently stray from a public right of way and encroach on private land.
Tom Platt, from the association, said: "What we're not clear about is how wide this will go and our concerns are that there could be unintentional consequences around things like wild camping, ability to protest, but also for walkers' rights more generally.
"This could be the thin end of the wedge for more draconian measures that would restrict people's ability to walk and access and enjoy the countryside."
However, Sarah Hendry from the Country Land and Business Association said: "[Our members] find that because it's not a criminal offence police tend not to intervene unless there's something very specific. Local authorities tend not to intervene either.
"So we think that strengthening the powers in this way and making this specific act of trespass a criminal offence will send that signal that it is something that police and the authorities ought to get involved in."
The Home Office said the focus was on intentional trespass and there would not be any adverse impact on people who just wanted to enjoy the countryside.
It said illegal sites caused distress for residents and the proposals meant they could be challenged and removed as quickly as possible.
A public consultation is under way, and is due to end on 4 March.