Robbie Wilmott's injury-time winner saw off Hereford as Newport recorded their fifth straight win to move up to third place in the Blue Square Bet Premier.
After trailing at half-time to the first of Josh O'Keefe's two penalties, the Exiles hit three second-half goals.
Midfielder Lee Minshull equalised with a header before another O'Keefe spot-kick put the Bulls back in front.
But Minshull again headed Newport level before substitute Wilmott snatched it in the third minute of stoppage time.
Hereford were actually awarded three penalties, all three of them conceded by Exiles defender David Pipe, but the home side only managed to convert two of them.
Pipe was adjudged to have fouled Marlon Jackson in the box for Hereford's first penalty on 41 minutes, only for Newport keeper Lenny Pidgeley to save Marley Watkins' spot-kick.
But, barely 60 seconds later, Pipe was adjudged to have handled and, after a change of penalty taker, this time O'Keefe made no mistake.
Newport levelled on 72 minutes when Minshull headed in Andy Sandell's cross but, just three minutes later, the Bulls won another spot-kick when Pipe was again penalised for handball and O'Keefe restored the home side's lead.
Minshull headed in his second with nine minutes left from close range before Willmott found the top corner, thumping home Newport's 93rd-minute winner from just inside the area.
Newport are now just three points behind leaders Wrexham, with two games in hand over their Welsh rivals.
Hereford, unsettled by the late withdrawal of 18-goal top scorer Ryan Bowman because of illness, are now 13 points adrift of the play-off places in eighth.
VIEWS FROM THE DRESSING ROOM
Hereford manager Martin Foyle told BBC Hereford & Worcester:
"We shouldn't have lost that. The players have done us proud, but we just did not get over the line. But Newport's more experienced players knew how to win. They've spent a lot of money. We can't compete with that.
"Should he (David Pipe) get sent off? The referee didn't even book him, not even for the blatant handball when he punched the ball for the second one. All these things add up.
"Ryan Bowman sent a text at a quarter past five saying he was ill. That's great! I could have done with knowing a bit earlier to give me chance to reorganise but today's players are different."
Newport County manager Justin Edinburgh told BBC Radio Wales:
"The players deserved all the applause they received because they could have let their heads drop and could have felt hard done by but, credit to them, they have gone right to the end and got themselves back in the game.
"We had to dig in. It was a derby with a fantastic atmosphere, and we had to come from behind twice and that is testament to the character and belief in the dressing room right now. We just edged it and just deserved it.
"Forest Green and Kidderminster lost,but it's more important how we came out of this game and to score as late as we did with a fantastic goal."
Scott Donnelly gives away a free kick for an unfair challenge on Marlon Jackson. Samuel Clucas restarts play with the free kick.
Assist by Andy Sandell.
71:46 Goal scored
Goal! - Lee Minshull - Hereford 1 - 1 Newport Lee Minshull grabs a headed goal from close in. Hereford 1-1 Newport County.
Corner taken short by Michael Flynn. A cross is delivered by Michael Flynn, Lee Minshull takes a shot. Stefan Stam gets a block in. Inswinging corner taken by Scott Donnelly from the right by-line, James Bittner makes a save.
Corner taken right-footed by Scott Donnelly to the near post, Kingsley James makes a clearance.
Chris Sharp is penalised for a handball. Scott Donnelly restarts play with the free kick.
Michael Flynn produces a cross, Headed effort from deep inside the area by Lee Minshull misses to the left of the goal.
Substitution Danny Crow leaves the field to be replaced by Robbie Willmott.
Booking James Bittner goes into the referee's book.
James Bittner fouled by David Pipe, the ref awards a free kick. James Bittner takes the free kick.
Andy Sandell delivers the ball, clearance made by James Musa.
Chris Sharp gives away a free kick for an unfair challenge on Andy Sandell. Lenny Pidgeley takes the free kick.
The official flags Ismail Yakubu offside. Indirect free kick taken by James Bittner.
Booking Caution for Joshua O'Keefe.
Unfair challenge on Lee Minshull by Kingsley James results in a free kick. Andy Sandell restarts play with the free kick.
The ball is swung over by Michael Flynn, Headed effort on goal by Lee Minshull from inside the penalty box misses to the right of the goal.
Substitution (Newport County) makes a substitution, with Scott Donnelly coming on for Rhys Griffiths.
Substitution Brian Smikle replaces Marley Watkins.
Marlon Jackson takes a shot. Save made by Lenny Pidgeley.
Ismail Yakubu takes a shot. Comfortable save by James Bittner.
Foul by Joshua O'Keefe on Byron Anthony, free kick awarded. Michael Flynn takes the free kick.
Andy Sandell takes a shot. Joe Connor gets a block in.
Andy Sandell fouled by Marley Watkins, the ref awards a free kick. Indirect free kick taken by Lenny Pidgeley.
Rhys Griffiths challenges Luke Graham unfairly and gives away a free kick. Stefan Stam takes the free kick.
The referee blows for offside against Christian Jolley. Indirect free kick taken by James Bittner.
Ismail Yakubu sends in a cross, Stefan Stam manages to make a clearance.
Marlon Jackson takes a shot. Save by Lenny Pidgeley.
Andy Sandell has an effort at goal from outside the box which goes wide right of the goal.
Joshua O'Keefe fouled by Michael Flynn, the ref awards a free kick. James Bittner takes the free kick.
Marley Watkins produces a cross, save made by Lenny Pidgeley.
The second half has started.
45:00 +3:29 Half time
Half Time The half-time whistle blows.
Andy Sandell takes a inswinging corner.
Samuel Clucas gives away a free kick for an unfair challenge on David Pipe. Direct free kick taken by Lenny Pidgeley.
Chris Sharp takes a shot. Blocked by Tony James.
Michael Flynn produces a cross, blocked by James Musa.
Rhys Griffiths concedes a free kick for a foul on James Musa. Free kick taken by James Bittner.
Shot by Rhys Griffiths. James Bittner makes a save.
Luke Graham gives away a free kick for an unfair challenge on Rhys Griffiths. Andy Sandell takes the free kick.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.