One of the key demands of a new group of Eurosceptic Conservative MPs is "not achievable", Foreign Secretary Philip Hammond has said.
The 50-strong Conservatives for Britain group wants Westminster to be sovereign over European Union law.
But Mr Hammond said a "unilateral red card veto" for the UK would "effectively be the end" of the EU.
Later, President Obama told David Cameron the US was "very much looking forward" to Britain staying in the EU.
The American leader was speaking at a meeting with the prime minister at the G7 summit in Germany.
Mr Cameron wants to renegotiate the UK's EU membership ahead of an in/out referendum by 2017.
Speaking on the BBC's Andrew Marr Show, Mr Hammond said he believed treaty change would be needed so that any future measures agreed on, such as restricting EU migrants' welfare entitlement, could be "sustained against judicial attack".
Asked how a "red card, backstop" could be achieved on EU law, he said: "If you were talking about the House of Commons having a unilateral red card veto, that's not achievable, that's not negotiable because that would effectively be the end of the European Union.
"What we are looking for is a system where a group of national parliaments could operate a red card."
On the BBC's Sunday Politics, Tory MP Steve Baker, who is leading Conservatives for Britain in the House of Commons, said a sovereign parliament was "a very modest demand".
Told this had been ruled out by Mr Hammond, he added: "In that case, we will have to have an 'out' campaign."
He added: "The cat's out of the bag here. We need the best possible deal for Britain and that does include a sovereign parliament."
Mr Baker, the Member for Wycombe, also said he was confident "at least another 50" Conservative MPs would join his group. While it styles itself as a supporter of the prime minister's renegotiation plans, it says it will push to leave the EU unless far-reaching changes are secured.
The new group also includes former ministers Owen Paterson and John Redwood.
"It's not a surprise that some Conservative MPs wish to campaign to leave the European Union; the big question is how many," Mr Baker added.
He also said a lack of a pre-referendum "purdah" period, during which there are restrictions on government announcements, could favour the Yes campaign to remain in the EU.
"Many of us are very concerned that these purdah rules are unconventional and appear to allow the government to spend unlimited amounts of taxpayers' money right up to polling day," he added.
Mr Hammond said the government wanted to engage in the debate "in a fair way" and would ensure public money was not allowed to be "inappropriately spent".
Former Conservative deputy prime minister Lord Heseltine said there would be "absolute chaos" if government ministers were allowed to fight on different sides of the EU referendum.
He told BBC 5 live's Pienaar's Politics: "You go to one minister who says black is white and you go to another minister who says white is black.
"You can't govern a country like that."
Asked about this scenario, Mr Hammond said: "Government ministers will be free to support the government line."
Former Conservative chancellor Ken Clarke warned his party against becoming too divided over Europe, telling BBC Radio 4's The World This Weekend: "The Conservative Party must have a debate, and we had a pretty silly debate during the John Major government which led to our defeat in 1997."
Mr Cameron has opened talks with other European leaders including German Chancellor Angela Merkel and Polish Prime Minister Ewa Kopacz - who both want to ensure the UK stays in the EU.
Speaking at the G7 summit in Germany, the PM said the British public, not "individual parties", would decide whether the UK stays in the EU.
Rafal Trzaskowski, Poland's secretary of state for European affairs, said those who would vote in the UK referendum must be told the truth about the consequences of leaving the EU.
He told the Observer: "If you say you can leave and still be part of the internal market and keep your second houses, that you will still be free to travel, that there will be no customs duties, and so on and so forth - but that you will not have to accept free movement of workers, and you will not pay into the EU budget, of course people will vote to leave, but this is simply not true."
Speaking on the Marr Show, Labour leadership contender Liz Kendall said she would always campaign to remain in the EU, saying membership was vital for security and the economy.
Andy Burnham, who is also bidding to become the party's leader, said he would establish a separate Labour pro-EU campaign if he won the contest, "to learn the lessons of Scotland's independence referendum".