The authority running the Lake District National Park has been accused of "violating its World Heritage status".
Campaigners say a "massive increase" in motorbikes and 4x4 vehicles has "profoundly changed the landscape".
They have accused the Lake District National Park Authority (LDNPA) of neglecting the fells and promoting the use of tracks by off-roaders.
The LDNPA said it would be "preferable if people did not take vehicles on these routes" but it was legal.
Head of park management Mark Eccles said: "All unsealed routes with public vehicular access rights, including the ones in the Little Langdale, Oxen Fell and Tilberthwaite area, are subject to the same laws as surfaced [tarmac] roads.
"We encourage users to behave responsibly on what can be vulnerable tracks to minimise environmental impact and respect other users."
Save The Lake District campaigners say the LDNPA has a statutory duty under the Environment Act 1995 to give greater weight to conservation in any conflict between it and recreational interests.
They are asking Unesco, which oversees the World Heritage status designation, to force the LDNPA to use its traffic regulation powers to keep off-roaders off unsurfaced tracks.
A petition demanding a ban has been signed by more than 4,300 people.
Although traffic regulation orders "cannot be ruled out", using them was a last resort and the authority had "no immediate plans" to do so, Mr Eccles said.