Blog article
See all stories »

When technology fails - a two minute guide

When you live in the North of England as I do, Hawaii seems like the perfect place to be at this time of the year. The idea of waking to warm sunlight and long days, and surrounded by tropical flowers and trees is a cut above the driving rain and sub-zero temperatures we’ve had recently. However, for 38 minutes earlier in the month Hawaii must have been one of the most terrifying places on Earth. Imagine going about your business and receiving the following text message:

Emergency Alert

Ballistic missile threat inbound to Hawaii. Seek immediate shelter. This is not a drill.

 

You probably saw the headlines in the news as this false alarm spread panic across the islands. It was a mistake by an employee at Hawaii's Emergency Management Agency during procedures that occur in a shift handover. According to the BBC, the message was reportedly sent despite an onscreen prompt requesting confirmation. Just over half an hour later another message was sent:

There is no missile threat or danger to the state of Hawaii.

 

After the event, the analysis revealed that the alert system did not allow for a correction to be sent quickly to mobiles. State officials said a "cancellation template" would be created to address the problem and also two people are now required to sign off the issuing of an alert. When technology fails it can cause untold and devastating consequences. Thankfully, no-one was hurt in Hawaii but many people will have endured an unprecedented level of stress.

In financial services, technology failures are rarely a matter of life and death. However, they can lead to irreparable damage to brands, financial loss and the end of careers. We all remember June 2012, when the Royal Bank of Scotland lost the ability to process payments as a result of an upgrade to its batch scheduling process. Overall the crash cost the bank £175 million, and the FCA said that the IT failure had affected more than 6.5 million customers in the UK over several weeks.

Tracey McDermott, Director of Enforcement and Financial Crime at the FCA summarised the situation quite succinctly:

"Modern banking depends on effective, reliable and resilient IT systems."

If you have read some of my previous articles then you’ll know I still find it amazing that people don’t prioritise testing. I always ask myself why financial institutions are investing so heavily in new payment systems without also investing in technology able to test them effectively. The Hawaii incident once again reminded me of the chaos that can be unleashed through insufficient time and effort spent during IT projects, and weak working practices.

We have produced a paper with Finextra which looks at the challenges that testing poses for financial institutions engaged in payments transformation. You can read the full paper here

 

10099

Comments: (2)

Shailendra Malik
Shailendra Malik - DBS Bank - Singapore 23 January, 2018, 03:44Be the first to give this comment the thumbs up 0 likes

Interesting point, but sadly it seems the IT systems deployed in public works or utilities segment haven't yet considered the basic ring fences put in place by Financial systems. A simple maker checker would've eased out the issue if what you explain the real cause of alert being sent to a production system.

I beg to defer from your comment that enough testing isn't done on the systems that go live in Banking domain. Yes, there have been mishaps and certain miss outs in the banking sector that has created headlines, but with process automation comes the ever so micro focus on testing automation and overall simulation that is possible with new generation Automatic testing systems. 

Sadly, with the new technology, the upskilling of existing IT workforce to adopt the changing tech is another concerns many companies struggle with, but I believe the emphasis on testing has increased more in current times than before.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 24 January, 2018, 12:04Be the first to give this comment the thumbs up 0 likes

IME, apps from banking are far more robust than those from any other industry barring, maybe, ecommerce and transportation.

It'd help if you could cite 2-3 examples of problems caused in banking systems that could've been averted by more / better testing. Memory serves, the RBS issue was caused by the bombing of an update in production despite the fact that the same update had been tested and found okay in staging. Not sure what kind of testing could've prevented such a disaster.

Not saying there's no scope for improvement of testing in banking but credit where credit's due. One clear area for improvement I see is improving the testing methods so as to reduce downtime caused by planned maintenance. I don't see the Amazons and Expedias of the world shutting down for planned maintenance very often and wonder why the HSBCs and HDFC Banks should.

Anthony Walton

Anthony Walton

CEO

Iliad Solutions

Member since

16 Oct 2007

Location

Leeds

Blog posts

27

Comments

14

This post is from a series of posts in the group:

Innovation in Financial Services

A discussion of trends in innovation management within financial institutions, and the key processes, technology and cultural shifts driving innovation.


See all