Robbie Wilmott's injury-time winner saw off Hereford as Newport recorded their fifth straight win to move up to third place in the Blue Square Bet Premier.
After trailing at half-time to the first of Josh O'Keefe's two penalties, the Exiles hit three second-half goals.
Midfielder Lee Minshull equalised with a header before another O'Keefe spot-kick put the Bulls back in front.
But Minshull again headed Newport level before substitute Wilmott snatched it in the third minute of stoppage time.
Hereford were actually awarded three penalties, all three of them conceded by Exiles defender David Pipe, but the home side only managed to convert two of them.
Pipe was adjudged to have fouled Marlon Jackson in the box for Hereford's first penalty on 41 minutes, only for Newport keeper Lenny Pidgeley to save Marley Watkins' spot-kick.
But, barely 60 seconds later, Pipe was adjudged to have handled and, after a change of penalty taker, this time O'Keefe made no mistake.
Newport levelled on 72 minutes when Minshull headed in Andy Sandell's cross but, just three minutes later, the Bulls won another spot-kick when Pipe was again penalised for handball and O'Keefe restored the home side's lead.
Minshull headed in his second with nine minutes left from close range before Willmott found the top corner, thumping home Newport's 93rd-minute winner from just inside the area.
Newport are now just three points behind leaders Wrexham, with two games in hand over their Welsh rivals.
Hereford, unsettled by the late withdrawal of 18-goal top scorer Ryan Bowman because of illness, are now 13 points adrift of the play-off places in eighth.
VIEWS FROM THE DRESSING ROOM
Hereford manager Martin Foyle told BBC Hereford & Worcester:
"We shouldn't have lost that. The players have done us proud, but we just did not get over the line. But Newport's more experienced players knew how to win. They've spent a lot of money. We can't compete with that.
"Should he (David Pipe) get sent off? The referee didn't even book him, not even for the blatant handball when he punched the ball for the second one. All these things add up.
"Ryan Bowman sent a text at a quarter past five saying he was ill. That's great! I could have done with knowing a bit earlier to give me chance to reorganise but today's players are different."
Newport County manager Justin Edinburgh told BBC Radio Wales:
"The players deserved all the applause they received because they could have let their heads drop and could have felt hard done by but, credit to them, they have gone right to the end and got themselves back in the game.
"We had to dig in. It was a derby with a fantastic atmosphere, and we had to come from behind twice and that is testament to the character and belief in the dressing room right now. We just edged it and just deserved it.
"Forest Green and Kidderminster lost,but it's more important how we came out of this game and to score as late as we did with a fantastic goal."
Live Text Commentary
Full Time The referee signals the end of the game.90:00 +5:29
Luke Graham gives away a free kick for an unfair challenge on Robbie Willmott. Michael Flynn takes the direct free kick.90:00 +4:26
Scott Donnelly takes a shot. Save made by James Bittner.90:00 +4:03 Booking
Booking Robbie Willmott is shown a yellow card.90:00 +2:37
Assist on the goal came from Michael Flynn.90:00 +2:37 Goal scored
Goal! - Robbie Willmott - Hereford 2 - 3 Newport Goal scored by Robbie Willmott from the edge of the area to the top left corner of the goal. Hereford 2-3 Newport County.89:31
Unfair challenge on Andy Sandell by Luke Graham results in a free kick. Free kick taken by Lenny Pidgeley.88:22
Christian Jolley produces a cross, clearance made by Joshua O'Keefe.86:55
Andy Sandell produces a cross, Kingsley James makes a clearance. Free kick awarded for an unfair challenge on James Musa by Christian Jolley. James Bittner takes the free kick.86:43
The ball is crossed by Andy Sandell, clearance made by Chris Carruthers.86:10
The ball is sent over by Scott Donnelly, clearance made by James Musa.85:28
Centre by Brian Smikle, Byron Anthony manages to make a clearance.84:00
Centre by David Pipe.83:12
Unfair challenge on Byron Anthony by Brian Smikle results in a free kick. Lenny Pidgeley restarts play with the free kick.81:50
Joshua O'Keefe concedes a free kick for a foul on Lee Minshull. Andy Sandell takes the free kick.81:21 Substitution
Substitution Marlon Jackson leaves the field to be replaced by Chris Carruthers.80:21
Assist on the goal came from Andy Sandell.80:21 Goal scored
Goal! - Lee Minshull - Hereford 2 - 2 Newport Lee Minshull scores a headed goal from inside the six-yard box. Hereford 2-2 Newport County.78:47
Scott Donnelly fouled by Brian Smikle, the ref awards a free kick. Direct free kick taken by Michael Flynn.78:30
David Pipe takes a shot. Stefan Stam gets a block in.
Byron Anthony fouled by Chris Sharp, the ref awards a free kick. Lenny Pidgeley takes the free kick.76:08
The ball is sent over by Michael Flynn.74:46 Goal scored
Goal! - Joshua O'Keefe - Hereford 2 - 1 Newport Placed penalty scored by Joshua O'Keefe. Hereford 2-1 Newport County.73:53
Handball by David Pipe.73:10
Scott Donnelly gives away a free kick for an unfair challenge on Marlon Jackson. Samuel Clucas restarts play with the free kick.71:46
Assist by Andy Sandell.71:46 Goal scored
Goal! - Lee Minshull - Hereford 1 - 1 Newport Lee Minshull grabs a headed goal from close in. Hereford 1-1 Newport County.70:21
Corner taken short by Michael Flynn. A cross is delivered by Michael Flynn, Lee Minshull takes a shot. Stefan Stam gets a block in. Inswinging corner taken by Scott Donnelly from the right by-line, James Bittner makes a save.69:54
Corner taken right-footed by Scott Donnelly to the near post, Kingsley James makes a clearance.69:38
Chris Sharp is penalised for a handball. Scott Donnelly restarts play with the free kick.67:10
Michael Flynn produces a cross, Headed effort from deep inside the area by Lee Minshull misses to the left of the goal.66:06 Substitution
Substitution Danny Crow leaves the field to be replaced by Robbie Willmott.63:10 Booking
Booking James Bittner goes into the referee's book.62:52
James Bittner fouled by David Pipe, the ref awards a free kick. James Bittner takes the free kick.62:40
Andy Sandell delivers the ball, clearance made by James Musa.61:56
Chris Sharp gives away a free kick for an unfair challenge on Andy Sandell. Lenny Pidgeley takes the free kick.60:35
The official flags Ismail Yakubu offside. Indirect free kick taken by James Bittner.60:09 Booking
Booking Caution for Joshua O'Keefe.59:30
Unfair challenge on Lee Minshull by Kingsley James results in a free kick. Andy Sandell restarts play with the free kick.58:47
The ball is swung over by Michael Flynn, Headed effort on goal by Lee Minshull from inside the penalty box misses to the right of the goal.57:44 Substitution
Substitution (Newport County) makes a substitution, with Scott Donnelly coming on for Rhys Griffiths.57:18 Substitution
Substitution Brian Smikle replaces Marley Watkins.56:25
Marlon Jackson takes a shot. Save made by Lenny Pidgeley.55:50
Ismail Yakubu takes a shot. Comfortable save by James Bittner.54:56
Foul by Joshua O'Keefe on Byron Anthony, free kick awarded. Michael Flynn takes the free kick.54:41
Andy Sandell takes a shot. Joe Connor gets a block in.51:09
Andy Sandell fouled by Marley Watkins, the ref awards a free kick. Indirect free kick taken by Lenny Pidgeley.50:42
Rhys Griffiths challenges Luke Graham unfairly and gives away a free kick. Stefan Stam takes the free kick.50:07
The referee blows for offside against Christian Jolley. Indirect free kick taken by James Bittner.49:41
Ismail Yakubu sends in a cross, Stefan Stam manages to make a clearance.48:14
Marlon Jackson takes a shot. Save by Lenny Pidgeley.47:31
Andy Sandell has an effort at goal from outside the box which goes wide right of the goal.45:50
Joshua O'Keefe fouled by Michael Flynn, the ref awards a free kick. James Bittner takes the free kick.45:30
Marley Watkins produces a cross, save made by Lenny Pidgeley.45:01
The second half has started.45:00 +3:29 Half time
Half Time The half-time whistle blows.45:00 +2:56
Andy Sandell takes a inswinging corner.45:00 +2:13
Samuel Clucas gives away a free kick for an unfair challenge on David Pipe. Direct free kick taken by Lenny Pidgeley.45:00 +1:55
Chris Sharp takes a shot. Blocked by Tony James.45:00 +0:48
Michael Flynn produces a cross, blocked by James Musa.44:26
Rhys Griffiths concedes a free kick for a foul on James Musa. Free kick taken by James Bittner.43:35
Shot by Rhys Griffiths. James Bittner makes a save.42:24
Luke Graham gives away a free kick for an unfair challenge on Rhys Griffiths. Andy Sandell takes the free kick.41:57 Goal scored
Goal! - Joshua O'Keefe - Hereford 1 - 0 Newport Placed penalty scored by Joshua O'Keefe. Hereford 1-0 Newport County.41:12
Handball decision goes against David Pipe.40:47 Penalty
A save by Lenny Pidgeley from penalty by Marley Watkins. Hereford 0-0 Newport County.39:55
David Pipe gives away a Penalty for an unfair challenge on Marlon Jackson.37:55
Foul by Danny Crow on Joshua O'Keefe, free kick awarded. The free kick is swung in left-footed by Stefan Stam, Michael Flynn manages to make a clearance.37:38
Chris Sharp takes a shot. Save by Lenny Pidgeley.36:10
Chris Sharp concedes a free kick for a foul on Ismail Yakubu. Free kick taken by Lenny Pidgeley.35:53
Lee Minshull handles the ball and concedes a free kick. Stefan Stam restarts play with the free kick.34:27
The ball is delivered by Samuel Clucas, Joe Connor takes a shot. Blocked by Tony James.34:27 Booking
Booking Andy Sandell booked.33:49
Andy Sandell concedes a free kick for a foul on Marley Watkins.33:07
Chris Sharp gives away a free kick for an unfair challenge on Ismail Yakubu. Lenny Pidgeley takes the free kick.31:56
Samuel Clucas produces a left-footed shot from deep inside the penalty box which goes wide of the left-hand upright.30:14
Michael Flynn fouled by Joshua O'Keefe, the ref awards a free kick. Michael Flynn takes the free kick.28:23
David Pipe fouled by Samuel Clucas, the ref awards a free kick. Direct free kick taken by Lenny Pidgeley.26:56
Foul by Joshua O'Keefe on Michael Flynn, free kick awarded. Free kick taken by Lenny Pidgeley.25:46
Samuel Clucas takes a shot from just inside the penalty box which clears the bar.23:43
Christian Jolley takes a shot. Save by James Bittner. Andy Sandell takes a shot. Rhys Griffiths is ruled offside. James Bittner restarts play with the free kick.21:55
Effort from the edge of the box by Christian Jolley goes wide of the right-hand upright.18:38
Corner taken by Andy Sandell, clearance made by Marley Watkins.18:06
Free kick awarded for a foul by Joshua O'Keefe on Michael Flynn. Free kick taken by Lenny Pidgeley.17:51
A cross is delivered by Marley Watkins, Michael Flynn makes a clearance.16:31
The ball is delivered by Andy Sandell, clearance by Samuel Clucas. The ball is crossed by David Pipe, Headed effort from inside the area by Christian Jolley misses to the right of the goal.14:55
The referee gives a free kick against Rhys Griffiths for handball. James Bittner restarts play with the free kick.13:34
The assistant referee signals for offside against Chris Sharp. Lenny Pidgeley takes the free kick.12:53
Stefan Stam takes a shot. Save made by Lenny Pidgeley.11:51
Christian Jolley takes a shot. Save by James Bittner.11:21
The ball is sent over by Marley Watkins, clearance made by David Pipe.7:34
James Musa concedes a free kick for a foul on Rhys Griffiths. Strike on goal comes in from Andy Sandell from the free kick, save by James Bittner.6:55
Chris Sharp challenges Tony James unfairly and gives away a free kick. Lenny Pidgeley restarts play with the free kick.5:55
Tony James gives away a free kick for an unfair challenge on Joshua O'Keefe. Direct free kick taken by Kingsley James.1:37
Lee Minshull challenges Joshua O'Keefe unfairly and gives away a free kick. James Bittner takes the direct free kick.1:14
Andy Sandell produces a cross, Luke Graham manages to make a clearance.0:00
The match gets underway.
Live data and text provided by our data suppliers