Fears have been raised that plans to house only sex offenders in all three prisons on the Isle of Wight could damage the island's image.
The proposals affecting Parkhurst, Albany and Camp Hill are being discussed by local governors and prison service bosses.
Councillor David Pugh said he was "genuinely worried about the potential impact on the image of the island".
The island's independent monitoring board said no decision had been made.
Mr Pugh said: "All of us who live here on the island understand our prisons are very secure and they provide much employment for local people and we have many of these offenders already.
"But people are understandably concerned.
"This is only a proposal at this stage and certainly we're going to be talking to the prison service about how they would see it working in practice and how we manage the message around it.
"Ultimately it is a decision for them.
"One of the things we would seek reassurances from the prison service about, if they did implement such a change, would be to make sure these offenders were released into their own communities."
'Speed up rehabilitation'
Albany already houses mainly sex offenders and about 50% of Parkhurst's population are sex offenders, but Camp Hill only has a few.
Nigel Hawley, chairman of Isle of Wight Prison's Independent Monitoring Board, confirmed discussions were taking place.
He said: "[The plans] make sense really. Prisoners as they progress through the system come down from a category A, B, C and so on.
"Albany and Parkhurst are high security (category B) prisons so as people progress through the system to a cat C prison, at the moment we have to transfer them to the mainland which means we have to wait for a space.
"To be able to transfer them within our own estate as it were, would save all that waiting and it would speed up the process of rehabilitation - and of course it's a bit of a saving to the taxpayer as well."
He said it was "highly unlikely" that offenders would end up being released into the community on the Isle of Wight.
Mr Hawley added: "The only people who are released on the island are people whose addresses were on the island at the time of conviction.
"All sex offenders have to go to addresses which are approved by the probation service."
Most sex offenders finish their sentences in a category D prison, which would mean they would be moved from the island before release.
The Ministry of Justice said there were discussions ongoing about the future of the island's prisons but would not comment further.