Up to a third of England's colleges may have to merge or work closely together in a federation in order to survive, principals are warning.
Up to 120 could be too financially unstable to survive in their present form, it is thought.
The warning comes ahead of changes to the way colleges are funded and as cuts filter down to the front line.
Ministers said the deficit had tied their hands financially but promised to set colleges free from costly red tape.
The warning comes from Lynne Sedgemore, the head of the 157 Group of larger, high-performing colleges.
She said there were fears right across the sector but that smaller colleges felt more vulnerable.
"There's something about critical mass - being a certain size, economies of scale - that helps.
"Colleges are looking at all the options and these are mergers, federations and management buy-outs."
In terms of the amount of mergers and federations thought likely, she added: "The figure of 100 to 120 colleges is being mentioned across the sector."
With about 350 colleges in England, this represents about a third.
This figure is roughly in line with a KPMG report commissioned by the Skills Funding Agency (SFA) which said up to 100 colleges were deemed potentially too financially fragile to survive.
As the power to commission where learners are taught passes from the former Learning and Skills Council to local councils, many college principals fear competition from private providers and academies, as well as the loss of students living in other local authorities.
Some of the report's details have been shared with the BBC News website, but the SFA has refused to publish the report in full.
Ms Sedgemore said that college principals were shocked by the extent of the cuts in the first round and that they needed to be prepared for the future.
Maggie Scott, of the Association of Colleges, said there were some real fears councils would seek to keep their own students at colleges and schools in their own local authority area.
More sixth forms
She said: "It is a particular concern in London where the boroughs are actively trying to keep their students by setting targets for keeping more young people close to home.
"If one authority does that then the neighbouring authority will try to do that too.
"That's against choice and it has never been the aim of government to restrict choice."
Richmond-upon-Thames College in south-west London has planned to reduce its student numbers by as many as 40%, from 4,185 last year to 2,400 over the next few years.
This has resulted in plans for more than 80 job losses. Staff have agreed a series of rolling strikes as part of their efforts to prevent the cuts.
Ms Scott said colleges were facing increased competition from private providers and schools that were being encouraged to create small sixth forms.
Plans for a rapid expansion in academies, many with new buildings and a wide range of courses, also presented a threat to college sustainability.
However, her organisation is less pessimistic about the future than the 157 Group.
An AOC spokesman said college income would decline in the next few years.
"We may see fewer colleges or we may see smaller colleges."
He said colleges may also expand to take work on from schools, universities and private training providers who move into other areas.
Apprenticeships, part-time higher education, courses for 14-year-olds, international and privately-funded adult education were all possible areas of future income growth, he added.
But Ms Sedgemore warned: "One fear is that the cuts will be so draconian that it will damage us in a way that is unsustainable.
"What we are saying is work with us to stage them so it's in a way that's workable - don't hit us with a 25% cut. It has to be planned, it has to be staged.
"We have to start thinking differently in the light of all these changes - new sorts of confederations.
"There are bound to be less colleges moving forward," she warned.
Business Secretary Vince Cable said the public debt had tied the government's hands but he pledged to "set colleges free from costly bureaucracy and controls administered by organisations that form the current alphabet soup of further education quangos".
"These changes will make it easier for colleges to decide their own priorities and ensure they can benefit from our decision this week to refocus a further £150m spending to boost much needed apprenticeship places by 50,000," he added.