The sale of new diesel cars that do not meet latest emissions standards will face a one-off tax increase in April.
It will be levied on all diesels that do not meet the Real Driving Emissions Step 2 standards on emissions for the first year of ownership.
According to experts, it means that most new diesels would be subject to the rise.
Chancellor Philip Hammond said the tax change would apply only to cars, and "white van man" was unaffected.
Different rates of Vehicle Excise Duty will be levied according to a car's CO2 emissions band.
A Ford Fiesta or Vauxhall Astra would see a one-off £20 rise and a Land Rover Discovery a £400 increase. Cars in the top band, such as a Porsche Cayenne, would be hit with a £500 tax.
The chancellor said: "Drivers buying a new car will be able to avoid this charge as soon as manufacturers bring forward the next-generation cleaner diesels that we all want to see.
The move was part of a series of Budget policies designed to improve air quality and promote electric vehicles.
The chancellor also unveiled a £220m Clean Air Fund, and £400m - split equally between the Treasury and motor industry - to improve the charging infrastructure for electric vehicles.
There will also be another £100m in subsidies to help persuade consumers to buy electric vehicles.
Analysis: Richard Westcott, BBC transport correspondent
The key thing... if you've already got a diesel car, you won't pay more.
That's hardly a surprise, bearing in mind people were encouraged to buy diesels some years ago. The government wasn't about to slap a big tax on drivers who parted with lots of money in good faith.
From April though, if you are buying a new diesel, you will probably pay more tax in the first year. It depends on the emissions test that it had to pass, so I'd ask the dealer before you buy.
The new tax rise will apply until around 2021, by which time all new cars have to meet the tighter pollution rules. And this only applies to cars, not vans, trucks, etc.
So, it's more of a soft, brushing nudge rather than a big push to persuade people away from polluting diesels.
Of course, there is a danger that it convinces drivers to keep their old, dirtier diesels, rather than buy a new, cleaner one.
The UK's motor industry trade body, the SMMT, said the chancellor's diesel tax changes risked sending out mixed messages.
Chief executive Mike Hawes said: "Diesel buyers will not face any additional taxation for the next six months, but thereafter, will face additional charges which will undermine fleet renewal efforts, which are the best and quickest way to address air quality concerns.
"Manufacturers are investing heavily in the latest low emission technology. However, it's unrealistic to think that we can fast-track the introduction of the next generation of clean diesel technology which takes years to develop."
But Peter Williams, of the motoring group RAC, said: "The chancellor has chosen to be relatively light touch when it comes to taxing new diesel cars.
"Any new diesel car registered from 1st April 2018 will be hit with a higher first year tax rate unless they conform to the latest real world driving standards.
"So current beleaguered owners of diesel cars can breathe a sigh of relief that they will not be punished further by the Treasury - but they will need to keep their eyes on local authorities who may be introducing clean air zones in the near future."
However, he added that a side effect of the Budget announcement might be a risk that drivers will be encouraged to keep their older diesel vehicles.