A group of victims is complaining to the police ombudsman about failures to investigate reports in 1996.
Father Malachy Finnegan, who died in 2002 was accused of sex abuse by 12 people.
Some parishioners are refusing to set foot in the parochial house where some of the abuse happened.
Priests will no longer stay overnight in the parochial house after Saturday evening mass.
The late Fr Malachy Finnegan, a former teacher, worked in St Colman's College in Newry from 1967 to 1976 and was later the President of the school.
Victims claim that the police in Newry were alerted to the allegations in 1996 but failed to interview the priest who died in 2002. The police say that a formal complaint was never made but they did receive a report of historical abuse.
Parishioners have told the BBC Spotlight team that they will no longer set foot in the parochial house in Hilltown and that priests will no longer stay overnight there.
Recently the Catholic diocese of Dromore settled one of the claims against Fr Finnegan.
Bishop of Dromore, John McAreavey, said the abuse was "abhorrent" and admitted he made an "error" by officiating at Fr Finnegan's funeral in 2002.
The school began to remove the priest's image from its photographs last year.
That decision was taken in October, when the school's Board of Governors were informed that the Diocese of Dromore had reached a settlement with one of the 12 victims.
Fr Finnegan was never prosecuted for sexual abuse, but allegations against him were reviewed by the National Board for Safeguarding Children - a clerical abuse watchdog set up by the Catholic Church.
In a statement, the Bishop of Dromore said that during an independent audit in 2011, he "specifically" asked the watchdog to examine the cases involving Fr Finnegan.
The diocese said it was aware of 12 allegations against the priest. October's settlement refers to just one of those cases.
"The first allegation against Malachy Finnegan came to light in 1994 some seven years after he left St. Colman's College," the bishop said.
"The second allegation came in 1998 and was not related to his tenure at St. Colman's. No further allegations emerged until after his death in January 2002."
The wrong decision
Bishop McAreavey added that after speaking to victims, he realized that his decision to officiate at Fr Finnegan's funeral in 2002 was "the wrong one".
A a statement was published on St Colman's College's website saying that its Board of Governors "condemns in the strongest possible terms the physical, sexual and emotional abuse inflicted by Malachy Finnegan when he was in the employment of the College over 30 years ago".
The governors said they were "devastated that any pupil who was entrusted to the care of St. Colman's College should ever have suffered abuse".
"When informed in October 2017, that a case had been settled by the Diocese of Dromore, the Board of Governors instructed that Malachy Finnegan's image be removed from photographs which were on display in the College," the statement added.