A police investigation into allegations of anti-Semitism within Labour would not have happened if leader Jeremy Corbyn had been tough enough in "stamping it out", Lord Hain has said.
The former Welsh Secretary said anti-Semitism had become a "curse" in the party.
He said it would be difficult for Labour to "command cross community support" on the issue.
Labour said it was fully committed to defending the Jewish community.
It emerged on Friday that the Met Police had launched a criminal inquiry into allegations of anti-Semitic hate crime.
It comes after LBC Radio obtained what it said was an internal Labour document detailing 45 cases, involving messages posted by members on social media.
Speaking to BBC One's Sunday Politics Wales programme, Lord Hain said he was "very disturbed" by the investigation, "because I think it highlights what has been a curse in the Labour party over this past period".
"We've always been the party that champions individual liberty and human rights, and I find it deeply deeply upsetting and painful that anti-Semitism and anti-Semitic attacks have got to the point where the police are investigating it in the Labour party," said the former MP for Neath.
"That's the sort of thing you expect from fascists and racists and anti-Semites, you don't expect it from members of the Labour party."
He said: "If the party leader had been tough enough in stamping it out... this particular investigation I don't suppose would ever have occurred."
"There is a particular problem of a particular small group on the hard left in the Labour party, who seem to think that anybody standing up for their particular view in the Israel-Palestinian conflict, if they take a view that is seeming to defend the Jewish interest in that conflict, then they are targets," Lord Hain said.
"I do not believe this has actually assisted the Palestinian cause. In fact, the reverse.
"Undoubtedly this has cast a really dark stain over the party.
"I think if you go into a general election with such an important section of the community, the Jewish community, which has historically been majority Labour... actually against the party, I think that makes it very very difficult electorally."
He denied he was "having a pop" at Mr Corbyn but he added: "I think it will be difficult for the Labour party to get out of all of this in a way that commands cross community support again on this particular issue."
'Tackling anti-Semitism top priority'
In September, Labour's ruling body agreed to adopt in full an international definition of anti-Semitism, after a long-running row about Mr Corbyn's handling of the issue.
It vowed to incorporate all the 11 examples of anti-Semitism cited by the International Holocaust Remembrance Alliance into its code of conduct. But Jewish groups criticised an accompanying statement which the party said was aimed at protecting free speech.
A Labour spokesman said: "Jeremy Corbyn and the Labour Party are fully committed to the support, defence and celebration of the Jewish community and its organisations.
"We have updated our rules and procedures to make it easier to take action against anti-Semitism and Jeremy asked General Secretary Jennie Formby to make tackling anti-Semitism her top priority."
Watch Sunday Politics Wales, BBC One Wales, 1100 GMT, 4 November