Berwick required extra-time to reach the second round of the Ramsdens Cup as they beat 10-man Livingston at Shielfield Park.
The hosts took the lead just before the half-hour mark when new loan signing Owen Ronald's corner was headed home by Ross Gray.
Marc McNulty had a golden opportunity to level things up from the penalty spot but cleared the crossbar with his effort from 12 yards.
However, the Livi striker made amends late on by heading home Martin Scott's cross to force an additional 30 minutes.
The visitors were reduced to 10 men four minutes after the restart when Jason Talbot was dismissed for foul and abusive language.
Rangers made the most of the advantage on 99 minutes when Josh Morris thumped home a fine strike from the edge of the box.
Ronald celebrated his debut with a fizzing strike into top corner minutes later before Simon Mensing converted a consolation penalty for Livingston after Dean Hoskins' challenge on McNulty.