James, 20, trailed 2009 champion Merritt by a distance coming off the bend and slipped into reverse as he meekly crossed the line in seventh in a time of 44.99.
United States' Tony McQuay took silver and Domican Republic's Luguelin Santos bronze.
"I was really hungry for this title and probably the hungriest man on that track," said Merritt, who served a 21-month drugs ban in 2009, was runner-up to James in 2011 and failed to qualify for the London 2012 final because of injury.
"In fact, I was starving because of the disappointment two years ago and in London last year," he added.
Men's 400m final analysis
Michael JohnsonEight-time world champion and BBC Sport expert
"It isn't a shock that Merritt won the race, it is the way he won the race and the way James lost it. Merritt took the race out so far and I don't think James didn't think he would be able to keep that pace. James didn't want to run too fast too early but had to, to keep up. Kirani had nothing left to give at the crucial moment."
"I was mentally and physically in the zone, out to prove that I am still a force in the event at this level. I lost a lot when I was banned but I never lost my will to win."
James said: "It just didn't work out for me. It is back to the drawing board, see what's going on. I didn't come into the Worlds expecting this, but you just move forward."
But the evening belonged to one athlete, the poster girl of Russian athletics and the greatest female pole vaulter in history.
Isinbayeva had said that she would retire after these World Championships but recently reversed that decision, claiming she would return to competition after having started a family.
Upon learning that she had won gold ahead of America's Jennifer Suhr and Cuba's Yarisley Silva, Isinbayeva sprinted into the stands to be embraced by the spectators.
The London 2012 bronze medallist
eventually made her way back to the track to attempt to beat her own world record but the evening was not to end with a 29th world record for the Russian.
Isinbayeva's gold medals
2004 & 2008
2005, 2007 & 2013
World Indoor Championships:
2004, 2006, 2008 & 2012
It mattered little, however, as the overjoyed Russian fans serenaded their favourite on a lap of honour at the Luzhniki Stadium in Moscow, with the two-time Olympic champion returning the favour by shouting "thank you" to her fans.
"I have plenty of emotions, they will only increase, not disappear," she said. "I didn't jump the world record - no biggie. The record is a bonus.
"I started jumping and screaming after 4.82m, after 4.89m I let all my emotions come out.
"By 5.07m I was just tired, but technically the attempts were good. I have two more competitions this season so I will try to break the record again.
"Now I'm happy. I'm the pole vault queen, the crown is mine."
In the men's discus final, Germany's
picked up his third World Championship gold, to go with his London 2012 Olympic title.
Harting's fourth attempt of 69.11m was enough to see off Polish rival Piotr Malachowski, who could only manage a best of 68.36m.
Aman claims gold in close 800m final
Estonia's Gerd Kanter took bronze with his throw of 65.19m.
The men's 800m final did not feature any of the London 2012 medallists and
The 19-year-old ran down American Nick Symmonds over the final 50m to claim Ethiopia's first medal at a distance below 5,000m at a major championships. Britain's Andrew Osagie came home fifth in 1:44.36.
Milcah Chemos Cheywa
of Kenya claimed gold in the women's 3,000m steeplechase in a time of 9:11.65 to finish ahead of compatriot Lidya Chepkurui and Ethiopia's Sofia Assefa.
Eilish McColgan finished in 10th for Britain in a time of 9:37.33.
won her first major title, finishing ahead of Canadian Brianne Theisen Eaton, who earned a silver medal to hang next to husband Ashton Eaton's decathlon gold.
World junior champion Dafne Schippers of the Netherlands charged down the home straight to finish third and secure the bronze medal.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.