Care must be taken in deporting foreign prisoners, says Grayling
Chris Grayling has said he is not prepared to reduce the number of foreign nationals in British jails at the expense of other countries' safety.
The justice secretary told MPs that it would be wrong to deport a rapist back to their country of origin if they were likely to be freed once there.
"What is costs us, it would cost somebody else," he said.
While cutting the 13,000 foreign prisoner population was a major goal doing so would not be easy, he warned.
The record of the coalition and its Labour predecessor in deporting foreign criminals has long been a subject of political dispute.
Appearing before the Commons Justice Committee, the justice secretary was asked by Conservative MP Nick de Bois what the benefit to the exchequer was of removing foreign prisoners and consequently freeing up prison places.
"It would have a huge impact on the budget," he replied.
Mr Grayling said he had a "very clear goal" to reduce the number of foreign prisoners but would not do this by potentially compromising the safety of people in other countries.
"It is not as easy as it appears. What is costs us, it would cost somebody else.
"What I am not in the business of doing is taking a rapist from a country in Africa, who is in a jail here, and saying 'you go back here and we are not worried if you are in jail or not.
"I don't care where they are. I don't want dangerous criminals walking the streets."
Before acting, he added, the UK must be sure that an offender's country of origin was willing to imprison them for the same length of time as they would serve if they stayed in domestic jails.
Dialogue with other governments over the issue was being "redoubled", he said.
But he added: "I do not want to mislead the committee or the house into saying I think it is possible to deliver transformational change quickly because it isn't."
Figures published in September showed there were 10,951 foreign nationals in UK jails and 1,855 whose nationality was unrecorded.
Labour say the number of foreign criminals being deported has fallen since 2010 and enforcement procedures are failing.
As part of efforts to facilitate deportations, the UK spent £3m last year on a scheme to improve prisons in Jamaica and Nigeria in order to send inmates in UK jails back to their home countries.