Why banks are likely to face more software glitches in 2013
The money men appear to be accident prone.
Recent months have seen banks and other financial services hit by a variety of software faults.
In January alone we have seen the Lloyds Banking Group's use of the Faster Payments System - designed to speed up cash transfers - hit by a glitch delaying wage and bill payments; a malfunction with PayPal's Instant Payment Notifications cause some users to face duplicate charges; and US stock exchange operator Bats Global Markets acknowledge that a "system issue" meant over 400,000 trades had failed to fulfil its best-price guarantee.
And these faults pale in comparison with last year's bungled software upgrade which disrupted millions of customers' accounts at NatWest, RBS and Ulster Bank; and the computer-controlled trading error at US market-maker Knight Capital that decimated its balance sheet.
So what's going wrong?
End Quote Lev Lesokhin Executive vice president, Cast
It is a struggle to see where faults in a system are most likely to lie”
The core of the problem is that the business software used by the institutions has become horrifically complex, according to Lev Lesokhin, strategy chief at New York-based software analysis firm Cast.
He says developers are good at building new functions, but bad at ensuring nothing goes wrong when the new software is added to the existing mix.
"Modern computer systems are so complicated you would need to perform more tests than there are stars in the sky to be 100% sure there were no problems in the system," he explains.
"Business software is becoming increasingly complex, composed of sub-systems written in different programming languages, on different machines by disparate teams.
"This means no single person, or even group of people, can ever fully understand the structure under the key business transactions in an enterprise. Testing alone is no longer a viable option to ensure dependable systems."Legacy issues
Mr Lesokhin adds the West's banking system is particularly exposed because it was the first to install computer systems and much of the sector was badly wounded by the credit crunch's knock-on effects.
"Tough financial times mean a squeeze on budgets and less effort spent on modernisation and quality assurance," he says.
Michael Lafferty, chairman of the research company Lafferty Group, says this problem particularly applies to the UK.
"There's been massive underinvestment in technology in banks - it seems to be the case that the whole damn thing is held together by sticking plaster," he says only half-jokingly.
"You hear stories of Cobol programmers being dug up and brought back from retirement after 20 years."
This is a reference to the fact many banking applications are still based on the Common Business-Oriented Language. The code dates back to 1959 and is unfamiliar to many younger developers.
Further complicating matters is the fact many banks have opted to buy in software from third parties, letting them slim down their own IT departments.
"When it's outsourced you can't make changes to the [bought] code," says Ralph Silva, the London-based vice president of Banking Strategy at HfS Research.
"You have to make changes to your own code - and that increases the risks.
"Previously there used to be lots of people involved who looked at that stuff, making sure it always connected. Now those are the people who have too much to do and don't have time to check everything."
There's a buzzword, coined by the American programmer Ward Cunningham, for the problems hidden in computer systems as a result of corners being cut : technical debt.
The idea is that IT bosses have allowed a certain amount of "unfixed" code to accumulate in order to roll out new facilities on schedule. But as the debt has grown, so has the risk of systems becoming "gummed up".
"Software is inherently difficult, and for developers who are dealing with systems which have been added to, cropped and changed over the years, it is a struggle to see where faults in a system are most likely to lie," says Mr Lesokhin.
"Take for example the software error at Knight Capital.
"Most IT applications carry around dead code - which lies dormant because none of the live modules are using it. When Knight Capital ran an update in its systems, some of the dead code was brought back to life, causing the system to spit out incorrect trades."
Debts are often best tackled by paying off the principal lump sum. In the case of "technical debt", Mr Lesokhin believes this involves committing resources to work out which parts of banks' dormant code are most likely to turn zombie to avoid potentially calamitous consequences.Tie-up troubles
As if matters were not difficult enough, there's another factor to consider - the many takeovers that have reshaped the financial sector.
From the outside, the industry may look more streamlined, but behind the scenes it's another matter, says Mr Silva.
"Because of the banking licences in the UK, when you bring two organisations together, the transition from two systems to one system can take up to 10 years," he explains.
"It takes a long time as it all has to be done by the book by the regulators' rules.
"There's one big bank in our country that has a total of 50 different mortgage systems as a result of history and mergers and acquisitions.
"That's insanity. It should have maybe one for retail and one for wholesale. But 50 is ludicrous... it hugely raises the danger."
The British Bankers' Association declined to comment on the issues raised. And while the experts find it easy to diagnose problems, they admit solutions are harder to find - particularly at a time when the financial industry is under pressure to roll out new banking apps and other online services.
The consequence may be that we have to be prepared for further software failures so long as the firms involved allow their technical debts to mount.