Detainees at Dover's Immigration Removal Centre are being held for "unacceptably long" periods of time, according to a report.
The Independent Monitoring Board also said the number of people held for more than a year while their asylum claims were processed had nearly doubled.
But it praised staff for the way they dealt with frustrated detainees.
The UK Border Agency said some people did not comply with the removal process which prolonged their stay.
Dover and Deal MP Charlie Elphicke said he was going to take the issue up with Immigration Minister Damian Green.
He said it was a matter of justice that detainees' cases were dealt with more quickly.
The report stated: "There has been little sign of progress in hastening the time taken to resolve complex cases resulting in unacceptably long periods of detention in some cases."
Other concerns outlined in the document, which covered the period between January and December 2009, were the amount of drug use at the centre, the length of visiting hours and the frequent closure of the library.
John Roper, Monitoring Board chairman, said: "There continues to be a high proportion of ex-foreign national prisoners, many of whom become increasingly worried about their indefinite detention.
"Coming from prisons and being unused to the less strict regime of an IRC [immigration removal centre] it was not surprising that instances of aggressive and volatile behaviour continued during 2009."
He told BBC Kent that in the year ending April 2010 the average length of stay per detainee was 136 days, 59 detainees had been held for six months and 20 for more than a year.
Mr Roper said for the 12-month period up to April 2007 no detainee was held for a year and only 30 people had been kept at the centre for six months.
A spokesman for the UK Border Agency said: "It is our intention to keep the length of detention to the shortest possible period.
"However, in some cases, detainees themselves prolong their own detention by failing to comply with the removal process, in particular by refusing to provide accurate and timely information about their nationality and identity to secure travel documentation."
He added: "Detention is reviewed on a regular basis to ensure that it continues to be justified and necessary.
"We have a duty to protect the public from those who may pose them harm, in particular those who have committed serious criminal offences."
In July the chief inspector of prisons Dame Anne Owers said conditions had deteriorated and the centre had become "more like a prison".