The UK Border Agency is to be abolished with its work returning to the Home Office, Theresa May has announced.
The home secretary told MPs "its performance was not good enough".
The UK Border Agency was formed in 2008 as an arms-length agency of the Home Office but Mrs May said this had led to a "secretive culture".
It will now be split into parts focusing on the visa system and on immigration law enforcement. Both parts will report directly to ministers.
It is the second time the UK Border Agency has been split in just over a year.
Last year, Mrs May announced that the UK Border Force - responsible for day-to-day operations - would stop being part of the agency and become a separate law-enforcement body.
Addressing the House of Commons, the home secretary said the improvement in performance of the Border Force since the split showed the benefits of having smaller, more focused structures.
She said: "But the performance of what remains of UKBA is still not good enough. The Agency struggles with the volume of its casework, which has led to historical backlogs running into the hundreds of thousands.
"The number of illegal immigrants removed does not keep up with the number of people who are here illegally. And while the visa operation is internationally competitive, it could and should get better still."
Mrs May said: "UKBA was given agency status in order to keep its work at an arm's length from ministers. That was wrong. It created a closed, secretive and defensive culture."
She said a board would be formed to oversee all the organisations in the immigration system - immigration policy, the Passport Service, Border Force and the two new entities.
The UKBA split would create "a high-volume service that makes high-quality decisions about who comes here" and an organisation that has "law enforcement at its heart".
In other changes, Home Office permanent secretary Mark Sedwill has been asked to revamp its "inadequate" IT systems and Mrs May said she would bring in a parliamentary bill by the end of the year in a bid to make it easier to remove illegal immigrants.
Her latest statement comes after fresh criticism of the agency's performance.
'Not fit for purpose'
MPs criticised the border agency in a report published on Monday, describing it as "not fit for purpose".
The Home Affairs Committee also said that for years the UK Border Agency had repeatedly supplied it with incorrect information about the size of the backlog of asylum cases.
Committee chairman Keith Vaz welcomed the home secretary's decision to put the border agency "out of its misery".
But in a heated committee session after Mrs May's announcement, the MPs were told by Mr Sedwill that the decision to axe the border agency had only been taken on Tuesday morning.
Mr Sedwill, the senior civil servant in the department, said the new structure would begin on Monday, and in a leaked memo to staff said most would "still be doing the same job in the same place with the same colleagues for the same boss".
The agency was created in 2008 by the merger of the Border and Immigration Agency, UKvisas and parts of HM Revenue and Customs.
This followed Labour Home Secretary John Reid's 2006 declaration that the Home Office's immigration directorate was "not fit for purpose".
The UK Border Agency is responsible for securing the UK border at air, rail and sea ports and migration controls, such as the issuing of visas.
It employs the full-time equivalent of 23,500 people across the world, including 13,100 in the UK.
'Made things worse'
Shadow home secretary Yvette Cooper said problems with immigration had got "worse, not better" since the coalition government came to power.
She told MPs: "The home secretary is right to say there are problems at UKBA and it has had a series of problems over many years.
"And we would have some sympathy with your proposals but the trouble with the proposals is you are simply refusing to recognise problems around enforcement and effectiveness at UKBA have got worse and not better on your watch."
She said: "We've been here before. You have already split the UKBA once.
"But what in fact has happened since last year's split? Queues at the border went up - Border Force presided over some of the longest queues our airports have ever seen.
"And things got worse at the Border Agency too - the select committee report showed a 20% increase in the backlog of asylum cases in three months."