"The issue is one that is vote-defining for many people."
Sitting in a café in Southgate, in North London, the Conservative MP David Burrowes looks worried.
He represents the marginal seat of Enfield Southgate and is chairman of the Conservative Christian Fellowship.
Mr Burrowes is a long-standing critic of gay marriage.
"Even in my constituency, where they know that I am firmly opposed to the proposal, people say that if that does go through as legislation, then they won't vote for me. That must concern me and it concerns many other colleagues in marginal seats."
The prime minister gave his support to same-sex marriage at last year's Conservative Party Conference when he said: "I don't support gay marriage despite being a Conservative; I support gay marriage because I'm a Conservative."
Critics of the idea hoped the government might just not get round to legalising it.
But David Cameron made a definitive promise last week. He told lesbian and gay groups at a reception in Downing Street that gay marriage would be legal by 2015.
Those remarks have re-energised a debate amongst some Conservatives.
Mr Burrowes' concern is simple. Those who hate the idea of gay marriage will not vote Conservative if it becomes law, he claims. But for those who like it, introducing it would not necessarily be enough to guarantee their support.
There appear to be three principal strands to Tory opposition to gay marriage: moral, religious and political.
"My concern is that we are potentially upsetting our members and activists when I have one goal, and that is to obtain a Conservative majority government in 2015. Anything which upsets any of my members, I don't like to see that," Emma Pidding, the Chairman of the Conservative National Convention told the BBC. The National Convention is the organisation which represents volunteers within the party.
The debate within the Conservative Party on gay marriage appears to reflect the debate in society at large. Some are passionately in favour. Some are passionately against. Many are relatively indifferent.
'Right side of history'
But critics of the plan are vociferous.
The Tory MP Mark Pritchard said he was waiting to see if Mr Cameron would make similar comments at this autumn's party conference to the ones he made in 2011.
"I think, in the desire to so-called 'de-contaminate' the Conservative brand, there are those that may end up damaging our Conservative brand with our natural supporters and voters in the country, and most importantly our grassroots who support us financially and deliver leaflets day-in, day-out up and down the country," warned Mr Pritchard. "We need those people and we shouldn't take them for granted."
But Matthew Sephton, the chairman of LGB Tory, an organisation for lesbian, gay and bisexual Conservatives, said opponents of same-sex marriage were a "vocal minority" in the party but did not represent voters at large.
He pointed to a recent YouGov poll which suggested 71% of the general public support the government's commitment to same-sex marriage.
"As David Cameron has said, we need to be on the right side of history and I think the government's proposals to introduce same-sex marriage will put us on the right side of history," he added.
David Burrowes, Mark Pritchard and Emma Pidding are all in favour of civil partnerships, but for Mr Burrowes and Mr Pritchard, gay marriage is a step too far and not a priority at a time of such economic uncertainty. Ms Pidding told the BBC she did not have a strong view on the issue either way, but acknowledged some fellow party activists did.
One MP, who asked not to be named, told me when addressing the AGM of their local party, being seen to be Eurosceptic was usually the best way of ensuring a big round of applause. But right now, condemning gay marriage was a better bet.
The government will respond to a consultation on gay marriage it set up earlier this year by Christmas.
Conservative MPs will be given a free vote on any legislation that follows, but the plans could become law regardless, with support from Labour and the Liberal Democrats.
But at what cost to harmony within the Conservative Party?