Robbie Wilmott's injury-time winner saw off Hereford as Newport recorded their fifth straight win to move up to third place in the Blue Square Bet Premier.
After trailing at half-time to the first of Josh O'Keefe's two penalties, the Exiles hit three second-half goals.
Midfielder Lee Minshull equalised with a header before another O'Keefe spot-kick put the Bulls back in front.
But Minshull again headed Newport level before substitute Wilmott snatched it in the third minute of stoppage time.
Hereford were actually awarded three penalties, all three of them conceded by Exiles defender David Pipe, but the home side only managed to convert two of them.
Pipe was adjudged to have fouled Marlon Jackson in the box for Hereford's first penalty on 41 minutes, only for Newport keeper Lenny Pidgeley to save Marley Watkins' spot-kick.
But, barely 60 seconds later, Pipe was adjudged to have handled and, after a change of penalty taker, this time O'Keefe made no mistake.
Newport levelled on 72 minutes when Minshull headed in Andy Sandell's cross but, just three minutes later, the Bulls won another spot-kick when Pipe was again penalised for handball and O'Keefe restored the home side's lead.
Minshull headed in his second with nine minutes left from close range before Willmott found the top corner, thumping home Newport's 93rd-minute winner from just inside the area.
Newport are now just three points behind leaders Wrexham, with two games in hand over their Welsh rivals.
Hereford, unsettled by the late withdrawal of 18-goal top scorer Ryan Bowman because of illness, are now 13 points adrift of the play-off places in eighth.
VIEWS FROM THE DRESSING ROOM
Hereford manager Martin Foyle told BBC Hereford & Worcester:
"We shouldn't have lost that. The players have done us proud, but we just did not get over the line. But Newport's more experienced players knew how to win. They've spent a lot of money. We can't compete with that.
"Should he (David Pipe) get sent off? The referee didn't even book him, not even for the blatant handball when he punched the ball for the second one. All these things add up.
"Ryan Bowman sent a text at a quarter past five saying he was ill. That's great! I could have done with knowing a bit earlier to give me chance to reorganise but today's players are different."
Newport County manager Justin Edinburgh told BBC Radio Wales:
"The players deserved all the applause they received because they could have let their heads drop and could have felt hard done by but, credit to them, they have gone right to the end and got themselves back in the game.
"We had to dig in. It was a derby with a fantastic atmosphere, and we had to come from behind twice and that is testament to the character and belief in the dressing room right now. We just edged it and just deserved it.
"Forest Green and Kidderminster lost,but it's more important how we came out of this game and to score as late as we did with a fantastic goal."