The show's ratings have been dropping in recent years and Simon Cowell said it could be because the acts aren't as good as they used to be.
Cowell is returning as a judge to this year's show and in a radio interview he talked about the popularity of the talent contest.
He said: "I think the show is a really good show but it really does depend on - not just who is on the panel - but who is actually competing on the show, if you look at the last year I did, it's the year we had One Direction, a very popular young girl called Cher Lloyd and a great singer called Rebecca Ferguson - they were all incredibly talented people."
Read more:What makes 1D a global success?
He added: "If you have people on the show that haven't got that world class potential - the audience will go down."
The pop mogul, who's been away judging the American version of the show, said the contests in recent years just haven't been as good as the show in 2010, when he last appeared as a judge.
The new series of X Factor is set to hit UK screens again in April and former judge Cheryl Cole will be back again.