A former senior intelligence official has questioned the impartiality of the committee scrutinising Britain's security services.
Sir Francis Richards said it was "not a very good idea" for an ex-Conservative minister to head Parliament's Intelligence and Security Committee.
But Sir Malcolm Rifkind said he was well placed to do the job as he had "handled intelligence" as a minister.
It follows questions about intelligence leaked by US whistleblower Ed Snowden.
Sir David Omand, a former UK homeland security adviser, has suggested the public disclosure of thousands of top-secret files by former CIA spy Edward Snowden was the most "catastrophic loss to British intelligence ever".
The episode has raised questions about the oversight of the British security services, with Business Secretary Vince Cable among those urging a review of current arrangements.
The Guardian - which published the original revelations about the data-gathering and surveillance activities of the US National Security Agency and its relationship with UK agencies - has suggested that a debate was starting in government about updating the way politicians oversee the security services.
The intelligence and security committee has access to classified intelligence material and holds its meetings in private for security reasons.
Reporting to Parliament, it publishes an annual review of the UK's intelligence policy as well as the administration and funding of MI5, MI6 and GCHQ.
Its remit was expanded earlier this year to cover operational matters, as well as the intelligence-related work of other government agencies such as the Joint Intelligence Committee, the National Security Secretariat, the Defence Intelligence department in the Ministry of Defence and the Office for Security and Counter-Terrorism.
Sir Malcolm, a former foreign and defence secretary under John Major, was appointed to chair the committee by Prime Minister David Cameron in 2010.
Sir Francis Richards, a career diplomat and former director of the GCHQ listening post, said he did not doubt Sir Malcolm's qualifications or ability to do the job but questioned whether he is "well-placed to command confidence".
Sir Malcolm told BBC Radio 4's World At One, that the committee was "completely bipartisan" and the views he expressed as chairman were those of the whole committee, which has members from all parties.
"I'm perfectly happy to criticise and attack the government if they do something silly or stupid," he added.
"Quite a number of people on the Intelligence and Security Committee are former ministers because we are people who've actually handled intelligence. We've had to deal with it as users of the product.
"We are responsible for deciding whether the security services do something dumb or stupid or illegal and if they do, we are prepared to be very very brutal with them."
Sir Malcolm also said those calling for more scrutiny "seemed to have forgotten" that the government had "dramatically increased" the powers of the committee.
No 10 has insisted there is no government review of oversight of the security services and the prime minister thinks the current system is working effectively.
But he said it was open to members of the National Security Council and the Privy Council to ask the security agencies questions, and he said David Cameron was prepared to listen to ideas from others.
Mr Clegg told the BBC that the ISC's powers had already been "significantly strengthened" but there was an "ongoing debate" about whether more needed to be done.
"And I think many of these revelations have quite rightly raised questions about how we can increase oversight further where that is clearly justified," he said.