Retailers have complained that Weymouth, which is hosting the London 2012 sailing events with neighbouring Portland, is a lot quieter than they expected it to be for the opening of the Games.
The local council predicted 60,000-plus visitors to the town each day, but despite the main spectator area being sold out the 15,000-capacity "live arena" on the beach is struggling to see more than 1,000 people at any one time.
Dennis Spurr, 59, of The Fantastic Sausage Company in Weymouth, said although the events organised in the town to celebrate the Games were "fantastic", it was the way the Games themselves had been promoted that was to blame.
"The Olympics have damaged my business," he said.
"It's the negative advertising - beat the delays, plan your journey now, come by train - people have been put off, they're not coming.
"Normally this time of year you can't move.
"The town is dead and businesses are really worried."
Mr Spurr admitted that the recent heavy rain and the recession had not helped business, but he said he still felt "really let down" by the dampened expectation of the Games in Weymouth and Portland.
"On Saturday I went home and I was so deflated - I'd been ready for the hype, I'd bought extra equipment, in my mind's eye I'd got queues across the street - but it didn't happen," he said.
Councillor Ian Bruce, from Weymouth and Portland Borough Council, said the traffic issues predicted for Weymouth and Portland during the Olympics "hadn't materialised".
"Traffic is flowing freely and we still have plenty of capacity within the park and ride sites," he said.
"Our message to people across the country is to come to Weymouth and Portland and enjoy a once in a lifetime opportunity."
James Parsons, 42, owns the Nothe Tavern, close to the ticketed Olympic site at Nothe Fort.
He said business was much quieter on the opening day of the sailing events than on a usual Sunday in July.
"We've had busier Sundays in November," he said.
"We're taking a significant loss at the moment."
Lee Fry, 50, from The Victoria Hotel, said road closures on the seafront had also been a problem during the Games.
"We had to turn two people away yesterday just because they couldn't park anywhere," he said.
"There's no passing cars and that's what we rely on for business - if people can't get to us, we're struggling."
Felicity Cranny bought Limelight clothing shop in Weymouth four years ago hoping the Olympics would pull in extra trade.
The 27-year-old said: "I thought the Olympics coming would be a great opportunity and yes things have been good on the whole, but just not this summer.
"I think a lot of people have been put off purely by the traffic and parking worries, but the roads are dead and the seafront has got plenty going on."
Ann Newman, 56, from Imagine Books in Weymouth, agrees there have not been the "hordes" of visitors business owners were expecting, but she said it was up to businesses to come up with ideas to draw people in.
"I've filled the front window with books on athletes, the Olympics and sailing, and although the buying hasn't really happened yet at least we've had the publicity and Weymouth has been put on the map," she said.