A quarter of employees think their company turns a blind eye to workplace bullying and harassment, according to a report by the Chartered Institute of Personnel and Development (CIPD).
Although 15% have experienced bullying in the past three years, more than half of them did not report it to the firm.
"Fear is the biggest factor," said one respondent. "You're singled out when something happens to you."
The CIPD urged firms to train managers to handle such conflicts better.
The report by the CIPD, which represents HR professionals, was based on two online surveys carried out by polling organisation YouGov.
One canvassed the views of more than 2,000 workers, while the other surveyed HR professionals and decision-makers.
The CIPD also conducted an online focus group with workers who had experienced bullying and harassment.
Some people said they suffered from stress, anxiety, insomnia, heart palpitations and suicidal thoughts.
"I had to have anti-depressants and counselling," said one worker. "I still can't go to the town where I worked because of panic attacks."
"It can take a lot of courage for someone to speak up about inappropriate behaviour at work, but there are very disappointing results on the ability of organisations to deal compassionately and effectively with complaints," the CIPD said.
"Many people felt their organisation didn't act swiftly or fairly to resolve the complaint, or that they were even being blamed for the situation."
The most common form of bullying or harassment was "being undermined or humiliated in my job", reported by 55% of women affected and 50% of men.
Then came "persistent unwarranted criticism" and "unwanted personal remarks".
Around 4% of employees said they had been sexually harassed over the past three years, the CIPD said. It described the problem as "stubborn", despite decades of equalities legislation.
But it said there had been "positive change" in the past two years in employees' willingness to stand up to sexual harassment, with 33% feeling more confident to challenge it.
One manager who took part in the survey said that some people did not report bullying or harassment because they were scared they might be "overreacting".
"I got told that on numerous occasions," the manager said. "Some people may not know who or how to report it."
Rachel Suff, senior employment relations adviser at the CIPD, said the survey was "a wake-up call to employers to put training managers at the heart of efforts to prevent inappropriate workplace behaviour".
She added: "Our research shows that managers who've received training can help to stop conflict from occurring and are much better at fostering healthy relationships in their team.
"And when conflict does occur, they can help to resolve the issue more quickly and effectively."
The CIPD added that firms should "encourage a speak-up culture" with a well publicised complaints procedure.
Managers should also be aware that such issues might sometimes best be resolved informally, especially if the behaviour was unintentional, it said.