Members of the West Midlands Police Federation said Andrew Mitchell had "no option but to resign", following a meeting with the chief whip.
They said Mr Mitchell repeated a "profound apology with feeling" but continued to deny using the words attributed to him during an outburst at police officers outside Number 10.
This meant his position had become "untenable", they said.
Shadow home secretary Yvette Cooper has called on Andrew Mitchell to be sacked.
Three representatives of the local Police Federation met Mr Mitchell for 45 minutes to discuss allegations the minister called Downing Street police officers "plebs".
Speaking to the media afterwards, Ken Mackaill, chairman of the West Mercia Police Federation, said Mr Mitchell refused to reveal what he actually said in the incident, apparently telling them he did not want a "fire fight with the police".
Prime Minister David Cameron and senior officers have said a line should be drawn under the matter following Mr Mitchell's previous apology, but pressure on Mr Mitchell continues.
A leader column in the Daily Telegraph on Friday called on the chief whip to step down from the cabinet saying he is now a "laughing stock" and a "walking, talking embodiment of everything with which David Cameron would least like his party to be associated".
According to an opinion poll by YouGov, 50% of the public think the "plebs" row shows Conservative MPs see themselves as better than ordinary people.
The online survey of 1,710 people suggests 60% of the public believe Mr Mitchell's outburst "showed his true feelings", while 50% thought his behaviour and attitudes reflected those of other Conservative MPs. Just over half of respondents, 52%, thought he should resign.
Mr Mitchell's future was the source of speculation during this week's Tory conference, despite the Sutton Coldfield MP choosing not to attend, with ministers repeatedly asked about the incident.
Deputy Prime Minister Nick Clegg told BBC Radio 5 live what Mr Mitchell had done was "really wrong" but he had apologised and he hoped the minister's meeting with the Police Federation would "clear the air".
Labour, which had previously called for the cabinet minister to provide further clarity on what he said to police officers, said the situation had gone on long enough.
'Pleb and proud'
"Neither the prime minister nor the chief whip have proved capable of coming clean swiftly and putting this right," said Ms Cooper.
"And it is now clear no one even in the Conservative Party has confidence in Andrew Mitchell either.
"The failure by David Cameron and Andrew Mitchell to take this incident seriously enough and to sort it out straight away means Andrew Mitchell will clearly not be able to instil respect in Parliament or beyond as chief whip, and this will just drag on and on.
"David Cameron needs to put an end to this now and remove Andrew Mitchell from his position as chief whip."
Last month, police officers protested outside Mr Mitchell's constituency office wearing T-shirt with the words "PC Pleb and Proud" printed on them and representatives have called for a full inquiry into the matter.