Roberts said he will not wear the T-shirt in protest at what he perceives to be the campaign group's lack of action in combating racism in football.
"I think everyone should be united. I don't know what point he's trying to make," said Ferguson.
Kick It Out facts
Kick Racism out of Football began in 1993
It became the more wide ranging anti-discrimination body Kick It Out in 1997
Its first Kick It Out week of action was held in 2001
In the season 2010-11, the organisation had an annual budget of £453,913
Of that, £330,000 came from the Football Association, the Premier League and the Professional Footballers' Association
It employs seven staff
"He really should be supporting all the other players who are doing it."
Arsenal manager Arsene Wenger added that if black players did not back Kick It Out, then their actions would make the campaign "not credible".
Kick It Out's annual awareness drive started on Thursday and runs until 29 October.
Players have traditionally worn shirts promoting the work of the anti-discrimination group, but Roberts has told BBC Sport he will not wear one.
Referring to the high-profile incidents involving John Terry and Luis Suarez in 2011, he said: "I find it hard to wear a T-shirt after what has happened in the last year."
Chelsea captain Terry was
banned for four matches and fined £220,000
by the Football Association for racially abusing Queens Park Rangers defender Anton Ferdinand in a Premier League match at Loftus Road on 23 October, 2011.
QPR manager Mark Hughes expects Ferdinand to show his support for Kick It Out by wearing an anti-racism T-shirt on Sunday before Rangers' home game against Everton.
"I fully expect everyone to wear the T-shirt," said Hughes. "I think Anton is going to wear [it]. I haven't been told any different."
However, Roberts, whose Reading team travel to Liverpool this Saturday, said he knew of other black players who were considering not wearing the T-shirts this weekend.
The 34-year-old added: "I'm totally committed to kicking racism out of football but when there's a movement I feel represents the issue in the way that speaks for me and my colleagues, then I will happily support it.
"I think people feel let down by what used to be called 'Let's Kick Racism Out of Football'. People don't feel like they have been strong enough."
But Ferguson added: "If everyone believes in it you should do it together. We shouldn't have sheep wandering off - I think it gives the wrong message.
"All the players are wearing it. I have only heard that Jason Roberts is different. But he is very different: he plays a game and is in the studio 20 minutes after it. That is a great privilege."
Caught in the crossfire
Are players right to blame anti-discrimination group Kick It Out for not doing enough to tackle racism in English football?
And Kick It Out chairman Lord Herman Ouseley also suggested Roberts was wrong not to wear a T-shirt.
"A lot of progress has been made in tackling racism, although much more needs to be done," he said. "But it's ridiculous to say that by not wearing a T-shirt you are going to change things."
Wenger added that there is less racism in the English game now than when he first joined Arsenal in 1996, while conceding that the fight against racism is "never won".
"Against stupidity the best way is information, knowledge of things and examples of behaviour, of attitudes," said Wenger.
"It is not only racism, black and white, it is against all kinds of insults we still have in the stadiums. We must fight more against it.
"You look at some faces when you walk around the pitch and what they shout at you is scary. That is, for me, racism.
"There is still a lot to do, but I think it's good that society fights against it."
Aberdeen and Wales goalkeeper Jason Brown told BBC Radio 5 live he supported Roberts's stance, arguing that Kick It Out and the football authorities had not done enough to tackle racism.
Football needs to move on from John Terry-Anton Ferdinand racism row - Hughes
"I support him 100%," Brown said. "I've had this stance since an incident when I was playing for Gillingham against Chesterfield a few years ago. After the lack of support I had from Kick It Out and the FA, I said I would never support them because I feel they don't want to help players in the lower leagues."
Lord Ouseley responded to Brown's comments.
"I don't know the specifics of the case Jason [Brown] refers to, but if he approached Kick It Out and he didn't get help, that shouldn't have happened," he said.
A statement from the Professional Footballers' Association, the players' union, read: "The strength of our organisation has always been in its unity and the involvement of ALL former and current players in supporting our anti-racism initiatives.
"Now is the time for us to reinforce our support for all such initiatives both in this country and abroad."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.