More than 50 public figures have added their names to a letter in the Guardian newspaper saying the Pope should not be given the "honour" of a UK state visit.
Authors Terry Pratchett and Philip Pullman and actor Stephen Fry are among those critical of the Vatican record on birth control, gay rights and abortion.
A Catholic spokesman said the prime minister's comments were an "excellent response to this open letter".
Pope Benedict XVI's visit to Scotland and England begins on Thursday.
Other signatories to the letter in the Guardian include: Professor Richard Dawkins, Ken Follett, AC Grayling, Stewart Lee, Claire Rayner, Lord Foulkes, Lord Hughes, Professor Steve Jones, Sir Jonathan Miller, Lord Taverne, Peter Tatchell and Baroness Turner.
They say the Pope "as a citizen of Europe and the leader of a religion with many adherents in the UK, is of course free to enter and tour our country".
But they say the Vatican has "been responsible for: Opposing the distribution of condoms and so increasing large families in poor countries and the spread of Aids; promoting segregated education; denying abortion to even the most vulnerable women; opposing equal rights for lesbian, gay, bisexual and transgender people; failing to address the many cases of abuse of children within its own organisation".
The letter goes on to say it has also resisted signing many major human rights treaties and has formed its own treaties with states which "negatively affect the human rights of citizens of those states".
It adds that they "reject the masquerading of the Holy See as a state, and the Pope as a head of state, as merely a convenient fiction to amplify the international influence of the Vatican".
The letter came after Downing Street released a video message in which Prime Minister David Cameron said the Pope would be offered a "very warm welcome" on his UK visit.
The prime minister, who will not be attending the events in London on Friday, will have a private meeting with the Pope on Saturday morning.
Father Christopher Jamison, Papal visit spokesman, said Mr Cameron's comments outlined "the many reasons why all political parties have supported the state visit of the pope".
He added: "The 21st Century is a religious century, and secular Britain is to be commended for being open to giving a public platform to religious voices in this country.
"The Catholic Church wants to build a multi-faith society in Britain. And the Pope will have important messages about the role that religious faith can play in civil society."
In the film, posted on the Number 10 website, Mr Cameron said the visit would be "a very special four days, not just for our six million Catholics, but for many people of faith right across Britain and millions more watching around the world".
Mr Cameron said: "It's a unique opportunity to celebrate the enormous contribution that all our faith communities make to our society and to celebrate their role in helping to build a bigger and stronger society."
Responding to the content of the letter, a spokeswoman for the Cabinet Office said the Pope was "regarded and respected as a head of state by the government and the Queen".
She said: "A state visit allows us to build positive relations with the Holy See."
She also highlighted comments by Mr Cameron in his video message that "we have incredibly important work to do together on fighting poverty and disease, on winning the argument to get to grips with climate change, and on promoting a multi-faith dialogue and working for peace across our world".
The final preparations for the Pope's four-day visit came as organisers admitted that the number of people who were set to take part in some of the events are less than first predicted.
The Catholic Church in Scotland has conceded that only 65,000 places have been allocated for Thursday's open-air mass in Bellahouston Park, Glasgow, less than the earlier 85,000 estimate.
Fewer worshippers are now expected at Sunday's open-air mass in Birmingham, where the Church says around 55,000 people are due to attend, down from the original estimate of as many as 80,000.