Substitute Tim Mouncey scored an injury-time winner to send Portadown back to the top of the Premiership on goal difference from Cliftonville.
Darren Murray volleyed into the roof of the net on the half hour but Douglas Wilson headed the Swifts level on 60.
Two minutes later Gary Twigg restored his side's lead with a header into the bottom corner but Jamie Tomelty saw his deflected shot find the net after 85.
Mouncey hammered home from close range to secure a precious win for his side.
Portadown had won the last four meetings between the sides and enjoyed the better of an unexceptional first half, with Murray's opener nudging Ronnie McFall's side in front.
The forward, back in the side after serving a suspension, had another effort ruled out for offside, while Peter McMahon blasted over.
The hosts, who were looking for their first home league win of the season, lost Dermot McCaffrey and Grant Hutchinson to injury.
After the break, Gary Liggett went close with a downward header which clipped the base of the post but Wilson found the net with his point-blank headed effort from four yards after Ryan Harpur's inswinging corner had been flicked on.
Twigg headed into the bottom of the net from Ross Redman's corner, after being left unmarked, to put the Ports ahead again.
Tomelty's shot over Miskelly looked to have earned the Swifts a share of the spoils but Mouncey's late intervention ensured maximum points for the visitors.