Chancellor Philip Hammond has rejected calls by Labour and some Conservatives to announce the end of austerity in his spring statement on Tuesday.
He is expected to unveil the smallest budget deficit since 2002, thanks to better than expected public finances.
But he told the BBC national debt was still too high, adding: "There is light at the end of the tunnel... but we are still in the tunnel at the moment."
Labour has urged him to end the "pain and misery" of public spending cuts.
Mr Hammond told the Andrew Marr Show it would be wrong to pour "every penny" into additional public spending.
He said it was "very important moment" to see debt starting to fall after growing for 17 continuous years.
But he said: "We should be very careful looking at single sets of figures - one quarter or two quarters - we need to look at what's happening sustainably in the economy."
The day-to-day deficit has been eliminated, it emerged earlier this month, two years later than former chancellor George Osborne had wanted when he set out in 2010.
The UK is now running a surplus of £3.8bn in its current budget - the money borrowed to fund day-to-day spending rather than long-term investment - according to the Office for National Statistics.
"We have a debt of £1.8 trillion - 86.5% of our GDP," Mr Hammond said. "All the international organisations recognise that is higher than the safe level."
Mr Hammond told the BBC that "we need to get our debt lower".
"I think most people in this country would be horrified to be reminded that we have £65,000 worth of public debt for every household in this country," he said.
The chancellor is due to announce a half-yearly update of Britain's public finance figures on Tuesday, but confirmed he would not be delivering any new tax or spending measures.
"If there is the flexibility and the space to do something, then we will decide in the autumn how we are going to use that," he said.
Economists expect the chancellor will announce borrowing is set to be around £7bn lower in 2017-18 than had been predicted, when he unveils the latest forecasts of the Office for Budget Responsibility.
Also appearing on Andrew Marr, John McDonnell, the shadow chancellor, called on the government to end its austerity programme.
He said recent economic figures were "not a matter for celebration" - and said the chancellor "should be coming into the real world".
He said Mr Hammond had "shifted the deficit onto the shoulders of NHS managers, headteachers and onto the shoulders of local government leaders.
"They're facing a financial crisis because of government cutbacks," he told Marr.
He called the government's plans for an NHS pay deal "miserly and mean-spirited".
"Pay is at the moment just about matching inflation, that's all," he added.
Mr McDonnell has called on the government to use its spring statement to end what Labour called an "immediate crisis" in public services.
But Mr Hammond vowed to "look at the numbers" in the autumn Budget.
He said: "Local authorities have done an incredible job in delivering efficiencies," adding: "Of course they're under some pressures".