Rangers fan Fraser McLaren scored the equaliser and came close to handing Berwick Rangers a surprise win as the Glasgow visitors misfired on the ground that doubles as a speedway track.
Berwick had just about edged the first half but fell behind in stoppage time when Andy Little fired home.
McLaren broke clear to find the far corner after 62 minutes and had another effort saved by keeper Neil Alexander.
Chris Townsley found the net for Berwick, but his effort was disallowed.
Andy Little celebrates after putting Rangers ahead at Shielfield Park
It leaves Rangers sitting fourth in Division Three, having also drawn their opening away game at Peterhead, with Berwick second from bottom.
Rangers' old ambition to play league football in England was realised in circumstances they would not have imagined until the club's summer of financial turmoil.
As they crossed the border to sun-kissed Shielfield Park, memories in the capacity crowd returned to that famous victory in 1967, when the Wee Rangers caused one of the biggest shocks in Scottish Cup history with a 1-0 victory over the men from Ibrox.
Former Hibernian midfielder Lee Currie's early late challenge on Ian Black showed that the current Wee Rangers were going to be no respectors of reputations.
Darren Lavery was unfortunate not to break the offside trap on a couple of occasions as Berwick matched a side playing in the top flight last season.
Rangers had handed a debut to ex-AEK Athens man Anestis Argyriou in defence alongside fit-again Ross Perry in place of Kirk Broadfoot and Chris Hegarty.
With players of that pedigree, it seemed strange to see Rangers resort to the kind of long-ball game more associated, perhaps unfairly, with their Third Division rivals.
The visitors were relying on set-pieces to threaten and took the lead from a free-kick flighted in by Black, who had shortly before had a powerful volley deflected just wide of the far post.
Lee McCulloch rose unmarked at the far corner of the six-yard box to head the ball back into the path of Little to fire home his seventh goal of the season from seven yards.
That failed to knock Berwick out of their stride and Phil Addison fired across the face of goal from 18 yards immediately after the break.
McLaren had been dropped to the bench after last weekend's meeting with Annan Athletic, but the Rangers-supporting striker made up for that disappointment by speeding clear to fire into the far corner moments after being introduced as a substitute.
The feet of Berwick goalkeeper Youssef Bejaoui prevented Francisco Sandaza from quickly restoring the advantage and McCulloch missed the target with a 16-yard drive.
McLaren skewed a drive wide when again put behind a Rangers defence looking increasingly uncertain.
The substitute was giving a defence marshalled by United States captain Carlos Bocanegra a torrid time and his 20-yard drive was touched over by Alexander.
There was still time for Townsley to head home from close range following a corner only to have it chopped off by referee Mike Tumilty for a mystery infringement.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.