Premier League referees are in "sheer panic" at how the video assistant referee system is being used, says former Chelsea striker Chris Sutton.
The league has brought in VAR this season to decide on goals, penalties, red cards and offside decisions.
But a number of high-profile incidents have been criticised, as has the length of time before a verdict is given.
Sutton said: "I was a fan of VAR but should we take the subjective calls out of it?"
Speaking on BBC Radio 5 Live's Monday Night Club, he referred to the guidance being given to referees by Mike Riley, head of the Professional Game Match Officials Limited, which manages match officials in English professional football.
"The referees have been told by Mike Riley to not use the [pitchside] monitor because it will slow everything," Sutton said.
"I was told at the briefing that the referee has to make the final decision, but he is not doing that any more and that's a massive issue."
There were a number of controversial VAR judgements over the weekend. Liverpool's Roberto Firmino had a goal disallowed at Aston Villa for being marginally offside, Watford won a soft penalty against Chelsea and Everton were denied a spot-kick against Tottenham after a lengthy review, even though the ball struck the arm of Spurs' Dele Alli.
Video assistant referees carry out their incident reviews from monitors at Stockley Park in west London, but former Celtic, Blackburn and Norwich forward Sutton feels the system is not working effectively.
"At Stockley Park there is sheer panic," Sutton added. "The worst decision for me was the Dele Alli handball where his arm was in an unnatural position - it took three minutes to make that decision.
"The referees have the opportunity to walk across to the monitor but are not using it. They are not taking responsibility; they are just scared.
"The referees need to step up. If I was reliant on an a system and a back-up team and they failed me again and again I would make my own decisions, go across to the monitor and trust myself - because that's what I've been trained to do.
"Kick the monitors out as they do not like using them."
The Premier League clarified why several of the decisions were made over the weekend, including on the Firmino goal. It also said that the decision over Alli's handball was not overturned by VAR because it was not a "clear and obvious error" by the referee - citing the pressure Alli was under from Everton's Yerry Mina when the ball struck his hand.
Chelsea boss Frank Lampard had a VAR decision go against his side on Saturday when Jorginho made minimal contact on Gerard Deulofeu but the penalty was given via VAR after on-field referee Anthony Taylor had said it was not a spot-kick.
"On the pitch on Saturday, there wasn't enough for it to be a penalty," said Lampard on Monday.
"I would work with it [VAR] at the moment and see how we improve. It is an open conversation. I think the clinical nature is a positive; it is the subjective ones we need to look at.
"The one thing we have to give the referees is that it is new. We all want better."
Speaking on Saturday after Liverpool's 2-1 win over Aston Villa at Villa Park, Liverpool manager Jürgen Klopp criticised the decision by the VAR official to disallow Firmino's goal for offside.
"It's not right that we sit here and talk about it and laugh about it. Managers get sacked over it," Klopp said.
"My analysts showed me the footage after the game and I didn't see it as offside. We just have to make sure the new system helps the game."