The government has downplayed claims that the department responsible for the Olympics could be scrapped.
Shadow culture secretary Harriet Harman told the Evening Standard she had heard the Department for Culture Media and Sport (DCMS) could be abolished.
A free-market economic think tank says closing it could save £1.6bn.
But DCMS officials dismissed the reports as "absolute rubbish" and Downing Street said it "did not recognise" the reports.
As well as the Olympics, the department's responsibilities also include organising the government's role in the Queen's Diamond Jubilee celebrations and the switch to digital television.
Ms Harman, who is also deputy Labour leader, said she feared Prime Minister David Cameron "is about to deal yet another blow to the arts by abolishing the DCMS altogether".
Writing in the Evening Standard she said: "There are well-sourced rumours in Westminster and the arts world that after the Olympics, the government will announce that the DCMS is no longer needed."
And on her website on Friday, she said the prime minister should offer a "categorical assurance" that he would not abolish the department, once the London Games are over.
A spokesman for the department said DCMS has a "very bright future", despite regular rumours of its demise, while a Downing Street spokesman issued a stong denial, saying Number 10 "did not recognise" the reports.
The Institute of Economic Affairs (IEA) claimed closing the department "could save the taxpayer around £1.6bn per annum".
Director General Mark Littlewood said: "If the government isn't seriously considering closing down the Department of Culture, Media and Sport, then it should start doing so now."
But a DCMS spokesman described the claims as "bizarre", adding that 98% of the department's funding went to other organisations such as the Arts Council England.
The spokesman said: "If we closed, free museum entry would end overnight."
DCMS is responsible for the final day of the Jubilee weekend, including the parade down the Mall and the Royal Family's balcony appearance, the spokesman added.
If all spending at the department was stopped, the IEA said, the savings could fund a 2% cut in corporation tax or a 3p cut in fuel duty.