A cruise firm boss said his passengers did not want to mix with "semi-lager louts" and lorry drivers "smelling of body odour" at a new terminal.
Lord Sterling, chairman of Swan Hellenic, was speaking at a visit to Portsmouth's new passenger terminal.
The life peer asked the port to provide separate facilities for cruise customers who will be sailing from Portsmouth from next summer.
He said his clients would not want to associate with ferry passengers.
Lord Sterling made the comments to representatives of the port as he was touring building work at the terminal.
Asked to clarify his remarks later, he said: "Many of our passengers are quite elderly and don't want people rushing around, or backpackers. On the whole they don't mix.
"It's not a question of segregation. What I'm saying is, in some ports you automatically have that as a facility where 'this is for cruise passengers' and 'that's for ferry passengers'."
Portsmouth City Council owns the commercial port and has invested more than £16m in it to try to attract new business.
Lord Sterling's specialist holiday company will base its three luxury liners in Portsmouth for the next five years.
The new public building, expected to be ready in February, will have airport-style check in desks, shops and restaurants and will be able to cater for 2,400 passengers.
Swan Hellenic's The Minerva, The Discovery and the Hebridean Princess will be the first cruise liners to use the new terminal and other cruise firms are set to follow.
The firm offers cruises to destinations such as the Mediterranean, Norway and Egypt starting from £1,550 for two weeks.
However, Portsmouth ferry port has until now mostly catered for cars and lorries using cross-channel ferries.
Harbour master Captain Rupert Taylor said: "We like to plan for all our customers, whether ferry or cruise ships.
"This new terminal has been built with this in mind, it's a lot larger and there are a lot more facilities where we can give private areas if necessary."
A statement from All Leisure Holidays, the parent company which owns the three liners, said: "Ferry travellers expect to embark and disembark as quickly as possible, whereas cruise passengers can arrive up to three hours before boarding and they expect a completely different kind of experience.
"For that reason, All Leisure Holidays is delighted that Portsmouth is offering a dedicated cruise area for passengers to enjoy."