Accrington Stanley defender Peter Murphy's late header denied Portsmouth a first victory of the season.
Murphy's curling shot had put the hosts ahead, before Connolly poked home after Patrick Agyemang's knock-down.
The veteran striker then slotted home from the penalty spot following Nicky Hunt's error, only for Murphy to nod in a far-post equaliser.
The goal meant Kal Naismith enjoyed an unbeaten start to his Stanley career, having signed on Saturday morning.
The 21-year-old agreed a two-year deal at the Store First Stadium following his release by Rangers.
Naismith, who has previously spent time on loan at Cowdenbeath and Partick Thistle, made 21 appearances for Rangers last season, scoring twice.
"It's great to be back," Naismith, who played for Accrington during a trial spell in pre-season, told the club's website. "I just need to come and play football, get away from Rangers and have the chance to work with the gaffer [James Beattie], who I know from Rangers.
"I need to play. I know the gaffer likes me as a player and I'll always work for him. Working with him and playing here will help me a lot."
VIEWS FROM THE DRESSING ROOM
Accrington Stanley manager James Beattie told BBC Radio Lancashire:
"I'm pleased, in the end, with a point. I'm very happy with the spirit and the togetherness of the group to come back into it.
"Peter Murphy was outstanding today. There were a few below-par performances and the lads know that themselves.
"It's a good result and it was a good game."
Portsmouth manager Guy Whittingham told BBC Radio Solent:
"We're disappointed that we haven't come away with three [points], of course we are, but having said that, it's a start.
"I think we played some good football, we were a lot more solid shape-wise and we broke off that and I think we did well.
"I think you look at the stats and look at how many chances we created and how many opportunities we had, you'd think you'd have to be more clinical. That's the one major disappointing thing out of it."