A "crisis" over same-sex marriage, Europe and attitudes towards its grassroots is threatening to destroy the Tory party, activists say.
In a letter delivered to Downing Street, 34 past and present local Conservative Party chairmen urged the PM to drop the same-sex marriage bill.
Reported comments from a senior Tory calling activists "swivel-eyed loons" were "no surprise", they added.
Downing St said it was "categorically untrue" anyone there made the comments.
Two national newspapers reported on Saturday that an ally of Prime Minister David Cameron had described local Conservative associations who put pressure on Tory MPs to vote for a Eurosceptic amendment to the Queen's Speech as "mad swivel-eyed loons".
Conservative co-chairman Lord Feldman said he was taking legal advice over "untrue" web rumours he had made "derogatory comments" and senior Conservatives rallied round him - Tory vice-chairman Bob Neill told Sky News: "I think we have to be very, very wary of this, I think, rather slipshod bit of journalism."
But Ben Harris-Quinney, director of Conservative Grassroots, said: "It doesn't matter who made these comments, the problem is that it comes as no surprise and is representative of a wider malaise in the party - the disconnect between the leadership and the grassroots."
He organised a letter and petition signed by 34 former and current local association chairmen, largely criticising the Marriage (Same Sex Couples) Bill for England and Wales, which returns to the Commons on Monday.
They complained of a "crisis of conservatism" and three issues which were currently "destroying the party" - Europe, same-sex marriage and the "contempt" for party membership from the leadership - as demonstrated by the "loons" comment.
"Your refusal to listen to reason and grassroots opinion is causing many previously loyal Conservatives to leave the party," they wrote.
"Some are lost forever and many will not contemplate re-joining unless the [same-sex marriage] Bill is abandoned or the party leadership changed."
They said the bill had led to voters switching to the UK Independence Party and claimed it made winning the next general election "virtually impossible".
On Europe, Mr Harris-Quinney said there was "nothing wrong with David Cameron's position on Europe as things stand" but both sides of the debate had been "angered by the process and lack of clarity in getting there".
"This is the problem in the Conservative Party now, there is no vision or narrative."
Bob Woollard, chairman of Conservative Grassroots, also told the BBC the same-sex marriage bill had "upset countless people and caused many, many people - hundreds, maybe thousands, to leave the party over this issue".
He said if there was no change it would be a "slippery slope downwards" with "more party members leaving the party, getting utterly disenchanted and frustrated that nobody is listening to them".
However, a different group of more than 100 Tory activists, including former Conservative Party National Convention chairman Paul Swaddle, later sent a separate letter to the prime minister urging the party to back the same sex marriage bill.
It said: "The majority of people in this country support same sex marriage, and many of them really do not understand the fuss.
"Faith groups are protected from the legislation should they so wish, therefore those of faith should not be worried by the Bill. This is an issue of particular importance to younger voters and MPs risk appearing out of touch if they pander to a vocal minority."
He added the part had to "deal with the Bill then move on together as a Party to outright Conservative victory in 2015".
Earlier, Health Secretary Jeremy Hunt said it was an "issue of conscience that is not really about party".
Mr Hunt told BBC One's Andrew Marr programme the prime minister was right to raise the issue even though it was "difficult" for many people.
"I personally support it," he said. "I support it because I believe in the institution of marriage and I think we should be encouraging people to make a lifelong commitment to each other. I think society is stronger if you do that.
"In my own case I got married in a church and not a register office because I happened to want to make my marriage vows in front of God.
"I think if gay people want to do that, and if the church is willing to conduct that ceremony, we shouldn't stand in their way."
And the Conservative former minister Nick Herbert told the BBC the country had "moved on" and attitudes towards gay people were changing.
Churches were protected, he said: "But this is a reform whose time has come."
He said not all Conservative activists were opposed to same-sex marriage and said the bigger danger for the party was losing touch with the next generation of voters.
"No political party, I think can survive unless it's in touch with the prevailing attitudes of the day."
Shadow Home Secretary Yvette Cooper told Sky: "I think it's a real problem if this gets lost in the vortex of the Tory infighting that we've had over the last couple of weeks, when actually it's a really positive bill that we should all want to celebrate."
But in a separate letter to Mr Cameron, Labour leader Ed Miliband and Lib Dem Deputy Prime Minister Nick Clegg, more than 500 Muslim community leaders and imams attacked the plans, saying marriage between a man and woman "is the cornerstone of family life, the only institution within which to raise children".
Under the bill, the Church of England and the Church in Wales will be banned from offering same-sex marriages because of their strongly stated opposition, unless they change canon law. Other religious organisations will be able to "opt in" to holding ceremonies.
There are currently no plans for similar legislation in Northern Ireland, but there are already plans for a bill to allow same-sex marriage in Scotland.