Attempts to topple Theresa May have been branded a "self-indulgent side show" by former minister Guto Bebb.
The Aberconwy Conservative MP said those writing letters of no confidence had "overestimated their power".
Theresa May, who has faced widespread criticism of her draft withdrawal agreement, will face a fight for her leadership if 48 members put in writing they have lost confidence in her.
To date 24 Tory MPs have openly submitted letters.
But some Tory Brexiteers claim the threshold has been reached but not all of the names have been made public.
"Even if they do get to the 48, the fact it has taken them significantly more time - and they might not even get there - is genuinely seen as an issue of some interest for my colleagues in Westminster," Mr Bebb told Radio Wales' Sunday Supplement programme.
"Since I left on Thursday afternoon the one issue we have been discussing in text messages and phone calls is the fact it looks as if perhaps Steve Baker, Jacob Rees-Mogg and others in the ERG group have actually overestimated their power."
Remain-supporting Mr Bebb resigned as minister for defence procurement to vote against the government on amendments it accepted to its Brexit Customs Bill in July.
He added: "My own reading of the situation is although it is highly unlikely that Theresa May will lose any vote of confidence - and she doesn't deserve to lose such a vote - to be perfectly frank it does look like a rather self-indulgent side show."
But while he said the prime minister was unlikely to face a vote of no confidence, she would struggle to get her draft withdrawal agreement through parliament because "the numbers in the house as they currently stand pose a real barrier".
Mr Bebb previously said the choice between the status quo and the prime minister's deal should be put to the public.