Labour and the Conservatives could face more resignations, with members of the new Independent Group saying they expect more MPs to join them.
Ex-Tory MP Heidi Allen told ITV's Peston programme "a third" of Tory MPs were fed up with the party's direction.
Tory MP Justine Greening said she would quit in the event of a no-deal Brexit, saying she couldn't stay in a party that became "simply a Brexit party".
MPs from the new group say they stand for "the centre ground of politics".
The group was set up by eight defecting Labour MPs unhappy about their party's handling of Brexit and anti-Semitism.
They were later joined by three pro-Remain Tories - Anna Soubry, Heidi Allen and Sarah Wollaston - who accuse the Conservative leadership of allowing right-wing hardliners to shape the party's approach to Brexit and other matters.
Chancellor Philip Hammond told BBC Radio 4's Today programme he was "saddened" by his former colleagues' comments, but hoped "over time they will feel able to rejoin the party".
He denied their claims that the pro-Brexit European Research Group, headed by Jacob Rees-Mogg, was now running the party, dismissing them as a "relatively small hardcore", and insisting the Conservative Party was still a "broad church".
Conservative MP and vice chairman of the European Research Group Mark Francois denied his group was a "party within a party" that had taken control, saying they were "a group of Conservative MPs who are passionately committed to honouring the democratic decision of the British people that we should leave the EU".
He added: "Conversely, Anna, Heidi and Sarah are now a party without a party, who are committed to precisely the opposite. As a result, we shall see how they get on."
Independent Group MP Sarah Wollaston has claimed a third of the cabinet would quit if there was a no-deal Brexit.
Mr Hammond refused to rule out his own departure - but he said the threat of a no-deal Brexit had "focused minds" and was encouraging compromise.
Justine Greening - a former education secretary - told the Today programme she tempted to break away from the party.
"It is something that I have considered, but I have reached a different conclusion for the moment," Ms Greening told Today.
"I don't think I would be able to stay part of a party that was simply a Brexit party that had crashed us out of the European Union."
Former Attorney General Dominic Grieve told BBC's Newsnight said he admired the courage of the breakaway group - and he would not be able to stay in the Conservative Party if it "went completely off the rails" and backed leaving the EU without a negotiated agreement.
Labour's Ian Austin also expressed sympathy with the Independent Group's aims, saying he would think "long and hard" about his future in the Labour party.
Shadow home Secretary Diane Abbott told BBC Radio 4's The World at One: "I am very sad that the Labour members of this new independent organisation have gone.
"Up until the last minute, people were talking to them, trying to persuade them not to take the step they have taken."
She said she hoped they would continue to work with Labour on issues like homelessness, the benefit system, the NHS and "most of all fighting this Tory Brexit".
Senior Conservatives have suggested the door is open for the three Tories who quit to return one day.
But Ms Soubry insisted there was no "going back" because the soul of the party had been overwhelmed by a rightwing "purple Momentum" a reference to former UKIP members they claim are joining local Tory parties to de-select Remain-supporting MPs.
She also revealed that former PM David Cameron had made a last-ditch attempt to stop the trio from quitting.
Speaking to The Times Red Box podcast, she said Mr Cameron had sent them a text saying: "Is it too late to persuade you to stay?"
Ms Allen also said she could not imagine returning "because if we do our jobs right there won't be a Tory party to go back to".
She said she was "hopeful" that what she described as "good, sensible centre ground colleagues" would join the new group.
Corbyn wants defectors to quit
Meanwhile, in a video released on Twitter on Wednesday night, Labour leader Mr Corbyn said defecting Labour MPs should resign and put themselves up for election.
He said this would be the "democratic thing to do" because they wanted to "abandon the policies on which they were elected".
Momentum, the Labour movement backing Mr Corbyn, is to hold "mass canvassing events" in several constituencies with Independent Group MPs to build support in the event of any by-elections.