Communities Secretary Eric Pickles has warned councils against using "under-the-counter pay-offs to silence departing staff".
Nearly 5,000 public servants may have been given pay-offs involving gagging orders, says The Daily Telegraph.
About 200 civil servants in Whitehall and 4,562 council workers have signed "compromise agreements", many of which involved confidentiality clauses.
The total cost over two years was £14m, according to the newspaper.
A Freedom of Information request found 256 councils in Britain signed compromise agreements with former staff between 2005 and 2010.
Mr Pickles said: "For too long, local government has made departing staff sign gagging orders, often with big pay-offs attached, away from the eyes of those who get left with the bill: the taxpayer.
"When leaving a job, councils and their employees need to part ways fairly.
"Giving out thousands in under-the-counter pay-offs to silence departing staff is not the way to achieve this.
"Councils have a responsibility to the public and transparency is at the heart of that.
"By shining a light on these activities and introducing new democratic checks and balances to stop gagging orders being abused we are helping councils improve accountability in local government."
Mr Pickles claimed new checks and balances would help stop gagging orders being abused.
But John Trickett, for Labour, said: "It is rank hypocrisy. Ministers are telling others to stop doing something which is rife in Whitehall."
The Department for Business, Innovation and Skills has signed compromise agreements with 83 officials over the past two years, at a cost of £2.6m, according to the Telegraph.
The Treasury has signed agreements with 64 individuals at a cost of £2.5m, although only a "small number" involved confidentiality agreements.
The Department for Transport signed 40 agreements in the past three years, all of which contain confidentiality clauses.
The Department for Energy and Climate Change signed 12 agreements containing confidentiality clauses at a total cost of £1.5m, the Ministry of
Justice signed 15 at a cost of £250,000. The Foreign Office spent £5.5m on compensation agreements.
Last month Health Secretary Jeremy Hunt banned the use of gagging orders that prevented NHS staff raising concerns about patient care in the wake of the Mid Staffordshire scandal.