Each bank holiday costs the UK economy £2.3bn and scrapping them would boost annual output by £19bn, economists say.
The Centre for Economics and Business Research (CEBR) think tank wants them to be more spread out over the year to stop businesses "losing momentum".
This year's extra bank holiday for the Diamond Jubilee means there are five in April, May and June outside Scotland, where Easter Monday is not a holiday.
Wales and England usually have eight, Scotland nine and Northern Ireland 10.
The think tank says that if bank holidays were scrapped, Britain's gross domestic product (GDP) - a measure of the value of goods and services produced by all sectors of the economy - would be £19bn higher every year.
It says the UK depends far more on services than other countries and that sector - with the exception of the hospitality industry - tends to work far less on public holidays.
CEBR founder Douglas McWilliams told BBC Breakfast: "About 45% of the economy suffers; the offices, the factories, the building sites where people tend not to go to work on bank holiday."
He said 15% of the economy, such as shops, pubs, restaurants and visitor attractions do well.
However, Mr McWilliams said that by spreading out public holidays, rather than scrapping them, people would enjoy them more.
Business can "lose momentum" when there are too many close together, he added.
But GMB leader Paul Kenny described the report as "utter rubbish", adding: "We could send kids down the mines again too and go back to working six days a week again as well.
"I'm not sure who would be in the shops, the restaurants and sports venues if we didn't have bank holidays."
British Retail Consortium director general Stephen Robertson told BBC Radio 4's Today programme the Easter holidays were good for shops, representing the start of the season for DIY and garden centres.
"We also begin to do outdoor activities, paint the house, and on top of that, if we have good weather then we have barbecues," he said.
"We socialise more and that's good for grocery shopping as well."
The CEBR points to South Korea, which has recovered rapidly from the financial crisis. Although there are more public holidays there, the think tank says different working conditions mean employee work over 500 hours more per year than British workers.
Unions have previously pressed for extra public holidays, pointing out that other European countries have more than the UK's minimum of eight.
Research published last year by Mercer HRsuggested there was a statutory minimum of 14 in Spain, 13 in Portugal, 12 in Greece, 11 in France, and nine in Germany and Ireland.
It found US and Australian workers get 10 public holidays, Canadians nine, Chinese 11 and Japanese 15. However, there are regional variations in many of these countries and employment laws differ as to whether workers should be paid for these holidays.
A fortnight ago the Governor of the Bank of England, Mervyn King, warned that GDP in the second quarter of this year might shrink owing to the number of bank holidays.