The Driving Standards Agency (DSA) area office in Cardiff, which employs 87 staff, is to close, it has been announced.
Caradog House, an administrative centre for the agency responsible for driving tests, will close in March 2012.
The DSA said not all jobs would go and it will consult trade unions about the possibility of staff being redeployed.
Union leaders described the closure as a "huge blow" while opposition parties criticised the decision.
Announcing the decision, Rosemary Thew, DSA chief executive, said that as the agency was funded by its customers' fees its duty was to spend that money responsibly, while maintaining a good level of service.
"That means being as efficient as possible in every area of our work and considering closely any areas of spending which may not be necessary," she said.
"Staff at Caradog House will be properly involved throughout the consultation process on the proposed office closure and we will be working hard to make sure that their questions and concerns are answered."
The DSA said a small office would be retained to support operational staff in the area.
Other responsibilities would be transferred to the agency's headquarters in Nottingham and northern area office in Newcastle.
The closure of Caradog House, in St Andrew's Place, near Cardiff city centre, would not affect testing provision in Wales, said the DSA, and the agency would continue to provide services under its agreed Welsh Language Scheme.
Public and Commercial Services union official Peter Harris said: "The decision is a huge blow, not just to the staff who face redundancy, but to the whole public sector and the economy of Wales.
"It comes after other announcements like the closure of the Passport Office in Newport, which represent a major attack on our public services and a threat to economic recovery."
Martin Ogilvie, the union's negotiations officer responsible for the DSA, added: "This is a short-term decision, which DSA management admit will save only £4m over 10 years."
Mr Ogilvie said the union would run a vigorous campaign to ensure its members were protected.
Shadow Wales Office minister Owen Smith claimed the UK government had held back on announcing the decision, which he said had been taken last December, "to get some distance between it and the slew of Welsh job losses and disinvestments they announced at the end of last year".
The Pontypridd MP said: "I think it is disgraceful that the government should have left workers at the DSA in limbo over Christmas, uncertain of their fate, only to save ministers from the difficult questions they will now have to face."
He said he would be writing to government minister Mike Penning and Welsh Secretary Cheryl Gillan asking them to explain precisely how and when the decision to close the office was made, and why the staff had not been informed promptly "as they ought to be under employment legislation."
Plaid Cymru transport spokesman Jonathan Edwards said: "This is dreadful news for 80 families facing a 'Blue Tuesday' as the Conservatives in London cut their jobs.
"As the major centre for Wales, the Cardiff Driving Standards Agency office also dealt with all Welsh language inquiries. What will happen to those facilities and jobs?"
But the DSA said there would be no effect on Welsh language services and they would continue to be offered as at present.
Liberal Democrat spokesperson Jenny Willott said she was "really disappointed" at the closure.
The Cardiff Central MP added: "While it is important that the government works to reduce public spending to help tackle the deficit, I believe it might have been possible to save more jobs, while still saving money, by moving the staff into vacant space in one of south Wales' many government-owned buildings."