The Scottish Police Authority, the body which oversees the national force, has reported a deficit of £34m.
Although this is a reduction from the previous budget, an Audit Scotland report warned that the SPA still faces considerable challenges.
The watchdog also expressed concern as to whether Police Scotland can meet its long-term strategic objectives.
SPA chairwoman Susan Deacon said the report noted "substantive improvements" in the last year.
But auditor general Caroline Gardner said a funding package for the force's new Data, Digital and ICT strategy - which is estimated to cost £298m over the next five years - has yet to be agreed.
As a result, the report identifies "a risk to both the timing of its implementation and the future financial sustainability of the Scottish Police Authority."
The report also reveals that a total of £4.3m was spent on consultancy fees in 2017/18 - more than double the £1.9m spent in the previous 12 months.
Expenditure on agency staff also soared to £4m, compared with £1.5m in 2016/17.
The report confirmed that Police Scotland's former chief constable, Phil Gormley, who resigned in February, received £28,227 for untaken annual leave and a further £54,137 payment in lieu of notice.
Ms Gardner said: "Policing in Scotland continues to go through considerable change.
"Progress has been made in key areas but there remains a substantial amount of work to do if the SPA is to achieve long-term financial sustainability and meet the challenges of modern policing.
"The scale, cost and complexity of the plans needed to deliver that transformational change should not be underestimated.
"It's vital that the SPA and Police Scotland develop comprehensive strategies for its future workforce, estates and ICT and clarify where the funding is coming from to make them a reality."
The report said seven new members had been appointed to the SPA and better budget monitoring had been put in place.
But it concluded that the national force, which replaced the old eight-force model in 2013, faced challenges to implement its Policing 2026 vision.
In particular it identified "slow progress in developing workforce and estate strategies" as factors that would restrict the SPA's ability to achieve long-term financial sustainability.
The combined Police Scotland and SPA annual budget for 2017/18 was £1.2bn but the total overspend was £34.3m.
British Transport Police
The report found that the SPA's three-year financial plan was designed to balance the books by 2020/21 by streamlining processes, reducing bureaucracy and introducing new technology.
And it noted members fees had increased from £0.17m to £0.24m as a result of additional time spent on transformation projects and covering for staff shortages at executive director level.
The auditor general also expressed concern over the proposed British Transport Police integration.
The original "go live" date of 1 April, 2019 was pushed back but a new date has yet to be set.
Consultancy support surrounding the move cost the SPA £300,000 and a significant amount of staff time.
The report said: "Given the range of transformational challenges facing the Scottish Police Authority and Police Scotland there is a risk that the ongoing uncertainty continues to absorb resources at the expense of wider strategic objectives."
Prof Deacon said major progress had been made in the last 12 months.
She said: "This year's annual report and accounts, and Audit Scotland's independent report, demonstrate that substantive improvements have been made.
"I am particularly pleased that Audit Scotland has noted that the authority now conducts its business in an open and transparent manner.
"This was, quite rightly, an area of significant critical comment in the past."
'Confidence and trust'
But the SPA chairwoman also said she did not underestimate the scale of the challenge faced by the SPA and the force.
Prof Deacon added: "I am confident, however, that we are now putting in place the people and the practices needed to build a police service which is fit for purpose, fit for the future and which commands high levels of public confidence and trust.
"I am determined to ensure that we work tirelessly to drive continuous improvement and so ensure that the people of Scotland receive the very highest possible standards of policing within available resource and that the governance and oversight of this service is robust and effective."
Police Scotland is the second largest police force in the UK after the Met and employs more than 22,000 officers and staff.