Thierry Henry's stoppage-time winner handed Martin O'Neill his first home defeat as Sunderland manager.
Arsenal fell behind after defender Per Mertesacker injured himself in trying to control the ball, with James McClean firing a shot past Wojciech Szczesny.
Mertersacker was carried off but his replacement Aaron Ramsey then directed a shot past Simon Mignolet.
But on-loan Henry, who returns to the New York Red Bulls next week, stylishly volleyed past Mignolet to grab the win.
It was a remarkable conclusion to Henry's second spell with Arsenal in the Premier League, even more so given he had come on as a substitute for Alex Oxlade-Chamberlain and had just the one chance.
HENRY'S SECOND ARSENAL SPELL
The 34-year-old has scored three goals in six games
The Frenchman showed why he has scored 176 goals in 258 top-flight games in those two spells as he cleverly directed his volley past Mignolet after ghosting in between John O'Shea and Michael Turner to connect with Andrey Arshavin's cross from the left.
Just as importantly Henry's goal enabled Arsene Wenger's side to leapfrog over Chelsea into the fourth Champions League spot on goal difference.
Wenger praises 'unbelievable' Henry
Such a scenario had looked highly unlikely given McClean had put Sunderland ahead with just over 20 minutes to go.
McClean took advantage of Mertesacker's misfortune after the German injured his ankle as he attempted to control the ball, with the Sunderland forward blasting a shot into the corner of the net.
Mertesacker had to be carried off but within five minutes Arsenal were level after another of their substitutes Ramsey picked up the ball on the edge of the box and guided his shot past Mignolet.
Henry's late intervention might have ensured the first defeat for Sunderland under O'Neill at the Stadium of Light, but what was evident was the enormous strides the Black Cats have made under their new manager.
Defeat tough to take - O'Neill
Sunderland expended enormous energy in trying to disrupt Arsenal's fluent style and, in McClean and Stephane Sessegnon, the home side had a couple of players with the speed and skill to always threaten Wenger's team.
Arsenal were also indebted to Szczesny who made a couple of good saves from Craig Gardner shots, while John O'Shea might have done better with a couple of second half-efforts.
In a scrappy first half, Sunderland had a strong penalty claim when the ball bounced off Mertesacker's hand.
Arsenal's best chance in the first period came from Theo Walcott, who flashed a shot past the post after good work from Oxlade-Chamberlain and Robin van Persie.
The two sides meet again in the FA Cup next Saturday, though at least Sunderland will not have to face Henry - who has now scored seven goals in 11 games against them.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.