Tory candidates for next year's Senedd election were asked how they would vote in a referendum to abolish the Welsh Parliament, as part of a party selection process.
It was one of three questions asked of all prospective Tory candidates for the South Wales Central regional list.
One party source dismissed it as "random", but another said it showed the party's "strong abolish streak".
The Welsh Conservatives declined to comment.
In November, Boris Johnson was quoted by The Sun to have told Tory MPs that devolution was ex-Labour prime minister Tony Blair's "biggest mistake".
But the Tories' Senedd leader Paul Davies said it was Labour-run governments that "have been a disaster for devolution in Wales".
The latest Welsh political barometer opinion poll suggests that 71% of people who intend to vote Conservative in the constituency vote and 67% who said they would vote for the party on the regional list would support abolishing the Senedd if there was a referendum.
As part of the Conservatives' shortlisting process for the South Wales Central list, candidates were given five minutes to make a speech and answer the same three questions.
As well as asking how they would vote in a hypothetical referendum on abolishing the Senedd, candidates were also asked about their priorities, and how as a regional member they would work with constituency members.
David Melding, who is standing down as Conservatives Member of the Senedd for South Wales Central at the 2021 election, said he did not believe "such a question would have been sanctioned by the Welsh Conservative Party".
"I fear that some in the party think the Abolish the Assembly Party's fox can be shot by such an ultra-sceptical tilt against devolution," he said.
However, one party source said of the questioning: "That's a question that comes up at various selections. This isn't a new question."
Another source said: "You can understand why people are asking it because we don't want to get out-flanked on this issue as we were with Brexit."
In total, eight candidates were shortlisted, including three spaces reserved for female candidates, and the current member of the Senedd (MS) and former party leader in the Senedd Andrew RT Davies automatically selected.
But Jonathan Morgan, who previously represented the party in Cardiff Bay and is currently an advisor to party leader Paul Davies, was not shortlisted.
Conservative MS for South Wales East Laura Ann Jones said on Twitter that she was "shocked and sorry" that Mr Morgan was not shortlisted.
Richard John, who is a Conservative councillor in Monmouthshire, said: "Really sorry to see this. Jonathan is one of the most talented Assembly Members we've ever had. Anyone without an agenda would agree he has earned a place on the ballot paper."
Really sorry to see this 😢 Jonathan is one of the most talented Assembly Members we’ve ever had. Anyone without an agenda would agree he has earned a place on the ballot paper https://t.co/YmfAvNUWOI— Richard John (@RichardJohnRJ) December 10, 2020
Another source told BBC Wales: "Think about who is chair of that regional area and who might've been threatened by a popular, pragmatic and demonstrably capable rival?
"If it looks like a stitch up, smells like a stitch up, walks like a stitch up, it is a stitch up."
Councillor Vincent Bailey, leader of the Conservative group on Vale of Glamorgan council, is chair of the party in South Wales Central.
Mr Bailey worked for Andrew RT Davies during his time as leader.
Jonathan Morgan said: "Having served for 12 years as a high profile AM and shadow minister I thought I had demonstrated the ability to get the job done.
"In the past nine years I've worked with a range of public services, I now chair a major housing association and advise the leader in the Senedd.
"We need people with experience of life outside politics as well as those with a track record," he added.
One Tory source criticised Mr Morgan for showing a "sense of entitlement", adding: "Only he is moaning - others aren't."