The Justice Secretary has defended his policy on books being sent to prisoners in England and Wales, through an open letter sent to the poet laureate.
Carol Ann Duffy protested over the ban on prisoners receiving parcels, including books, which began last year.
Chris Grayling said the ban kept prisons free of drugs, but Ms Duffy said it hurt the "soul of the country".
The Howard League for Penal Reform said drugs had not originally been cited as a reason for the ban.
In his letter, Mr Grayling said he had only specifically made a decision about "the availability of books in prisons" once, and that was to allow the novel Fifty Shades of Grey to be circulated in female prisons, in a bid to encourage reading.
He said that allowing prisoners to receive regular parcels would "put in jeopardy the efforts of prison staff to keep drugs and other illicit materials out of prisons".
"The arrival of thousands of unknown parcels in our prisons each day, whether containing books, essential items or anything else, would completely undermine these efforts," he said.
"It would be a logistical impossibility to check them all in the level of detail that is needed, to properly explore whether apparently innocuous items contain drugs or other illegal items."
Mr Grayling also denied he or any other ministers had made specific changes about the availability of books in prisons.
"Despite some reports, we have not sought to include them in a list of privilege items that have to be earned by offenders - to do so would be wholly wrong," he said.
He went on to offer Ms Duffy an invitation to visit a prison so she could "see in more detail the scale of the smuggling" in prisons, and the attempts being made to circumvent the rules.
But Andrew Neilson, director of campaigns at the Howard League for Penal Reform, described it as "bizarre" that a system which he said had worked perfectly well until November last year was now described by Mr Grayling as "logistically impossible".
He said concern about drugs had not been cited as a reason to impose the restrictions when they were introduced last year.
A National Offender Management Service document which heralded the restrictions last November says "the general presumption will be that items for prisoners will not be handed in or sent in by their friends or families unless there are exceptional circumstances."
The purpose of this is to "ensure that the IEP [Incentives and Earned Privileges] scheme is not undermined", it says.
"There are many ways that drugs get into prison, but few people would be stupid enough to pop them in the post," Mr Neilson said.
Ms Duffy organised the protest outside Pentonville Prison in north London on Friday, which was also attended by actress Vanessa Redgrave and author Kathy Lette.