Changing to the Alternative Vote (AV) for electing Westminster MPs will "bring in highly odd and more extreme people", Ken Clarke has warned.
The justice secretary, who is Conservative patron of the No2AV campaign, said electoral change "was not an effective protest vote".
The Conservative Yes2AV campaign said Mr Clarke's comments were "ludicrous" and AV would mean fewer extremists.
A referendum on change the voting system will be held on 5 May.
The Conservative Party has thrown its weight behind the No2AV campaign - and there are campaign posters all over the Swalec stadium, where it is holding its spring conference.
Most of the heavyweight speakers have urged voters to reject change in the 5 May referendum.
The far smaller Yes2AV Tory campaign has no official presence at the event - but it is making its voice heard at fringe meetings.
The No2AV campaign wants to mobilise Conservative voters to back its cause when they go to the polls on 5 May, with English local elections and Welsh and Scottish elections being held on the same day.
Speaking at the Conservative Party spring conference, Mr Clarke issued a warning to those thinking of voting for AV because they wanted to register a protest against politicians over recent "scandals".
He said it would open the door to fringe candidates who are just as likely to be corrupt as mainstream politicians as "sleaze is a cross-party issue" and would be even less representative of ordinary people.
"Electoral change is not an effective protest vote. Don't vote Yes to AV - you are going to get an even odder collection of politicians.
"The best way to hold politicians to account is to vote forcefully through our present system."
Matthew Elliott, chairman of No2AV, who took part in a panel discussion with Mr Clarke, said he was planning to launch a "No2AV yes to PR" section of the campaign next week.
This would be for people who felt "passionately" that the electoral system needed changing to a form of proportional representation but did not like the "miserable little compromise" of AV.
Instead of placing a cross in a box against one candidate, voters using the AV system rank the candidates in order of preference.
If no single candidate gets more than 50% of the vote, second, third and fourth preferences come into play.
Those in favour say it is fairer and means no votes will be wasted - but opponents say it will give more say to the supporters of fringe parties and extremist groups.
"You can be against changing the voting system to AV but still be in favour of voting reform," he told Tory activists.
Rene Kinzett, of the Conservative Yes2AV campaign said: "It is ludicrous to suggest that AV will lead to extremist candidates being elected.
"Candidates will have to build cross-party consensus and reach out into new areas of the constituency.
"They will no longer be able to be elected with the backing of three in 10 of their constituents."