Heritage campaigners have formally objected to a Her Majesty's Revenue and Customs (HMRC) plan to move thousands of workers into a Liverpool "landmark".
HMRC want to make India Buildings on Water Street one of its "modern, cost-effective regional centres" in 2019.
However, the Twentieth Century Society oppose HMRC's "harmful alterations" to the building and the removal of public access to a ground floor retail arcade.
A planning decision would be made at a later date, the city's council said.
Built in the 1920s and influenced by New York's architecture, the building once housed the Passport Office but has been largely empty in recent years.
The move is part of the HMRC's plan to close a number of offices and create 13 regional centres, the organisation's spokeswoman said.
'A great shame'
She said it would mean there will no longer be free public access to Holt's Arcade, the celebrated Art Deco retail corridor on the ground floor.
However, she said the organisation was "looking at how we can give other people the chance to experience this amazing space and will be working with the council on an appropriate solution".
Tess Pinto, conservation adviser for the Twentieth Century Society said the changes were unacceptable as India Buildings was "undoubtedly one of the finest inter-war buildings in the country and the shopping arcade is the centrepiece".
"Although it is not technically a right of way, the arcade has always been open for the public to enjoy and was designed for that purpose," he said.
"It is a great shame that a government body will be curtailing public access to this stunning space, as well as destroying a lot of the surviving fabric on the upper floors."
The building on Water Street was designed by architects Arnold Thornely and Herbert J Rowse for the Alfred Holt & Co shipping line and replaced an earlier India Building, which had stood since the 1830s.
Thornely also worked on one of the city's Three Graces, the Port of Liverpool building, while Rowse's other works include Liverpool's Philharmonic Concert Hall.
The building has provided the backdrop for several films and TV shows, with elements of it featuring in the 2001 movie, The 51st State, and the 2016 adaptation of JK Rowling's book, Fantastic Beasts and Where to Find Them.
Liverpool Mayor Joe Anderson said the proposed move "secures the future of one of our most-loved buildings".
He added that it was "another shot in the arm for our ongoing renaissance, which is seeing us attract investment, not just in new developments, but also in regenerating landmark buildings".