The annual levy banks pay on their balance sheets is to be reduced gradually but an 8% surcharge on their profits will be introduced.
The Treasury said the bank levy would be cut from its current 0.21% to 0.1% in 2021, by which time it would apply to banks' UK balance sheets only.
HSBC said earlier this year that the bank levy was a factor in whether to move its headquarters from the UK.
The new profit surcharge will start from 1 January next year.
The banking levy was introduced after the financial crisis as a way of forcing banks to contribute more to the UK economy, but also with the aim of discouraging them from risky borrowing.
Banks have pushed for the levy to be reduced or cut altogether, saying it puts them at an unfair disadvantage to their global competitors.
The levy particularly affects banks with large balance sheets, such as HSBC. Last year it paid £750m of the £1.9bn raised by the government through the tax.
Treasury estimates suggest the new profit surcharge and the changes to the levy will bring in about £1.7bn in extra revenue over the next five years.
The chief executive of the British Bankers' Association, Anthony Browne, said he welcomed the move to change the bank levy, but argued the new surcharge would "reinforce fears that Britain is becoming a less attractive place for banks to do business".
"We believe these moves will also undermine competition in the industry by making it harder for smaller players to break through and challenge larger banks," he said
PwC banking tax partner Matthew Barling agreed that the move sent a "mixed message" about the UK's competitiveness as a base for banks.
"The long-term phased nature of the reform coupled with the new profits based 8% corporation tax surcharge means that the overall tax burden on the banking sector will go up during this Parliament."
However, Anna Anthony, head of financial services tax at EY, said the move would be welcomed by the banking sector.
"The introduction of an 8% surcharge sounds high, but is likely to be more acceptable than the levy because it at least has a direct link to the profitability of an institution.
"In addition, the fact that overall corporation tax is going to go down also helps ease the pain for banks of the surcharge," she added.