One worker in 12 does not take the time off work they should, according to Trades Union Congress (TUC) analysis.
In the UK there are 1.2 million workers who receive no annual leave at all, and 2.2 million who get less than the legal minimum holiday of five and a half weeks a year, the organisation said.
Those missing out were "at risk of burn-out", said the TUC.
The government said employers were expected to treat workers fairly and abide by employment law.
In the UK most workers who work a five-day week are legally entitled to 28 days' paid leave per year. An employer can include bank holidays as part of that statutory leave entitlement.
Workers missed out on holiday either because they were set unrealistic workloads that could not be completed on time, or because employers were deliberately denying holiday requests and "managing out" people's leave, the TUC said.
But not granting leave caused stress and mental and physical ill-health, which had knock-on effects on co-workers, friends and families, the union body said.
The TUC suggests the tax authority, HMRC, should be given new powers to clamp down on employers who aren't giving staff the leave owed them.
A spokesman from the Business, Energy and Industrial Strategy (BEIS) said employers should make sure workers got the holiday pay and leave they were legally entitled to.
He said it was providing new rights to millions of workers, with holiday pay to be enforced for vulnerable workers and that these new plans would be outlined in due course.
People working in agriculture, mining and quarrying, and accommodation and food were the most likely to be missing out, according to the TUC's analysis.
It also showed:
- Workers are losing out on nearly £3bn worth of paid leave a year
- 9.2% of female workers and 7.2% of male workers are losing out
- The sectors with the highest numbers of staff losing out are retail (348,000), education (342,000) and health and social care workers (291,000)
"We're now in peak holiday season. But while many workers are away enjoying time off with friends and family, millions are missing out," Frances O'Grady, the TUC general secretary, said.
"Employers have no excuse for robbing staff of their well-earned leave. UK workers put in billions of hours of unpaid overtime as it is."
The analysis was based on unpublished data from the ONS Labour Force Survey from 2017.