Many special advisers are "glorified press officers", a former Cabinet minister has claimed.
Lord Crickhowell, Welsh Secretary under Lady Thatcher, told peers that many advisers existed to "feed" the media rather than to provide expert advice.
But Cabinet Office minister Francis Maude said their role was "much wider", encompassing more than party politics.
It emerged recently that the number of advisers has risen from 65 to 79 in the coalition government's second year.
Special advisers are employed as temporary government workers but do not have to be politically impartial like their civil service colleagues.
They link together the minister, the party and the department and can handle a range of functions such as writing speeches, devising policies or dealing with the media.
The issue of what they do arose during a hearing of the Lords Constitution Committee, which is examining the issue of civil service accountability.
Conservative peer Lord Crickhowell who, as Nicholas Edwards was Welsh Secretary for eight years in the 1980s, questioned the competence of some special advisers - known as spads.
"We discovered on getting evidence that there are an awful lot of people who are, what you may call, glorified press officers," he said. "They are there simply to feed the ever-burgeoning 24-hour press.
"The strongly held view was there was a need for really professional, skilled advisers in the area the minister is dealing with and may not have great experience of.
"And that really the balance ought to be changed so we have more special advisers who can provide real input into the work of ministers, independently of the civil service, rather than this large number of people who are there simply to give the current political view to Twitter."
But giving evidence, Mr Maude - who has responsibility for the civil service - said special advisers "came in all shapes and sizes" and while "there was a role" for people who could deal specifically with the media, most were "not in that category".
"In my experience, and the experience of most of my colleagues, that is a very small part of what special advisers do," he said.
Special advisers, he added, should be "highly intelligent", "unequivocally loyal" to their ministers and able to "scrutinise the work of mainstream civil servants".
He added: "I don't think the best use for a special adviser is to provide subject expertise. A department ought to be able to provide that and if they don't have it they should damn well get it."
Special advisers were there "not just to provide political input... but to ensure the work that comes to ministers actually reflects what ministers want, to help spot things that are wrong and to be very much on the case about whether decisions get implemented".
"It is a very wide-ranging role indeed and much wider than being just party political," he added.
The government revealed on Tuesday that the cost of employing special advisers has increased by about 25% in the coalition's second year.
In the written statement , Mr Cameron noted that the pay bill for the most recent year was still lower than the £6.8m spent in the final year of the last Labour government.