The government has described Northern rail services as "unacceptable" as delays continue to plague its trains.
Transport Secretary Grant Shapps said he was "not prepared" to tolerate the commuter line operator's poor performance.
He told the BBC: "There's a couple of ways that can go, but one is to strip the franchise."
Northern said the late delivery of big infrastructure projects had disrupted its services.
The RMT union described the comments as a "distraction" to draw attention away from rail fare increases coming into effect from Thursday.
In October, Mr Shapps said he had taken steps that could lead to Northern losing its franchise.
Giving evidence to the Commons' Transport Select Committee last year, the transport secretary said that he had issued a "request for proposals" from the company and the Operator of Last Resort.
This could mean Northern's service being brought under government control.
The rail operator company has faced several issues in recent years, such as cancellations after the introduction of new timetables.
In a BBC interview on Thursday, Mr Shapps was asked whether his intention was to remove Northern's franchise.
He replied: "That's right. In the autumn I wrote to the necessary parties in this with what's called a request for proposal, and that's simply where you say: 'Look, I'm going to take action.'
"There's a couple of ways that can go, but one is to strip the franchise, one is to have a short-term contract, but yes, exactly as you said, I'm simply not prepared for the service on Northern to carry on as it is and I am taking action."
David Brown, managing director at Northern, said the company had faced challenges outside of its control in recent years.
One includes the late delivery of major infrastructure projects, such as the North West electrification, more than two years late.
Mr Brown added: "Arriva and Northern remain fully committed to delivering the transformation of the North's railways and improving customers' experience. We are delivering the biggest transformation of local rail for a generation."
The RMT union's general secretary Mike Cash also called the announcement a "distraction" from fare rises.
He said: "If Grant Shapps was serious, he would set out a timetable for removing the Northern rail franchise from Arriva and their replacement with the public sector operator."
Millions of commuters now have to pay an average of 2.7% more for train tickets, with some annual passes to increase by more than £100.
What might happen to Northern?
The process to decide whether Northern should be stripped of its franchise is still ongoing.
The BBC understands a decision is likely to come in "a matter of weeks".
Broadly speaking, there are two options;
- The government could charge the Operator of Last Resort (a public company which already manages the LNER franchise) to take control of the Northern franchise. Under this scenario, Northern's parent company Arriva would be out of the picture
- Or it could convert the franchise into a "management contract", which would mean that Arriva would still be running the trains but for a set fee and the DfT would effectively set the parameters (i.e. what services, when, how long the trains etc).
The situation has arisen because Northern is not meeting many of its obligations under the current franchise contract (longer trains, new services (particularly on Sunday) but performance is a big part of it and the Northern franchise was losing money for some time.
Northern recently insisted its passenger numbers are growing again after a period of decline following the 2018 timetable.
It's unclear whether the franchise is still making a loss today.
A senior rail source told the BBC that there is certainly a "ramping up of the language" on the issue.
The Department for Transport confirmed that Mr Shapps had started the process that could strip Northern of its franchise.
Shadow transport secretary Andy McDonald said that Northern "should have been stripped of its franchise years ago".
Labour mayors Steve Rotherham and Andy Burnham previously called on the government to take control of the service.
A review of UK railways is already taking place.
It is being led by Keith Williams, the former boss of British Airways, and its findings are due to be published soon.