Blog article
See all stories »

Sealed with a glitch: Why are technical errors sneaking into payments systems?

Valentines day was sealed with a less than loving glitch for retailers this year. Thousands British businesses were left helpless after a technical fault hit their card payments system, on Valentine’s Day weekend.

The outage from Global Payments is the latest in a long line of IT glitches from the banking and payments sector. We’re less two months into the new calendar year, and already there have been system failures at Tesco Bank, Natwest and Sainsbury's Bank. 

With regulators due to report back on their investigations into the resilience of key technology infrastructure in the coming weeks, the financial services industry needs to ask itself the question: how are these defects creeping into production? Are financial service providers not testing thoroughly, acting in haste or is their infrastructure simply unable to cope? 

The payments industry has to find a way to move faster but not risk its critical reputational capital. To make sure that IT systems are resilient and reliable, there needs to be more focus on reducing the risks in testing. For example in the fast moving payments industry, test data can be obsolete after only a couple of hours. Even worse subsets of data or even synthesised data may be used, introducing greater risk of errors. Moving to a Data as a Service model means those organisations could develop and test using near-live data, thus ruling out glitches caused by data errors. Test environments can be restored in minutes rather than hours, increasing testing cycles by as much as 1000x.

The regulator has already shown that it’s prepared to take action against financial service providers that leave consumers financially stranded and businesses are increasingly intolerant of outages, mishaps and mistakes. However, demand for the latest and greatest services have never been higher. As the old Facebook developer mantra read, ‘Move fast and break things’, the urge to innovate increases the risk of mistakes. But if banks and payment providers are going to be able to move at speed and maintain stability, they need to be able to embrace new technologies and models that enable this agility.

Businesses cannot afford to have technical glitches, especially when they hit them where it hurts, the cash register. For retailers and traders, Valentine’s Day only comes once a year and once it’s gone that influx of sales is gone with it. Technical errors on days that are critical to the financial success of the business are a recipe for customer churn. With a greater focus on risk-free innovation, businesses can ensure they are moving at pace to provide the best possible service, in the most stable manner.


Comments: (7)

Stanley Epstein
Stanley Epstein - Citadel Advantage Group - Modiin 07 March, 2015, 10:39Be the first to give this comment the thumbs up 0 likes

You have hit the nail on the head when you suggest inadequate testing. I want to add a couple of extra points as well – bad design (usually because both the business and the technical requirements are not properly understood) exacerbated by the need for indecent haste in launching, demanded by the product owners within the institution. The need for speed is a very explosive factor that often turns sour.

A Finextra member
A Finextra member 09 March, 2015, 03:23Be the first to give this comment the thumbs up 0 likes

I'd certainly agree that a lack of focus on testing is a root cause. I'd qualify that further as an over reliance on manual testing techniques from the 1990s. Modern payment systems are too complex and change too fast for old fashioned testing techniques to be successful.

A Finextra member
A Finextra member 09 March, 2015, 08:46Be the first to give this comment the thumbs up 0 likes

Banks tend to treat developments as financial projects, so they do not plan for failure due to technical and social interaction in the build process.

On top of this the board is composed of narrowly financial people. And those are not alert to the type of disaster signals that technical projects show.

We call this disaster by optimism

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 09 March, 2015, 09:30Be the first to give this comment the thumbs up 0 likes

Not sure whether FPS, TARGET2 and SEPA qualify as modern payment systems but, in my experience of running their implementations, business always asks why automated testing isn't being used and IT always pushes back saying modern payment systems are too complex, change too fast and are subject to too many adhoc moving parts driven by nebulous regulation for automated testing to work. If automated testing has now become flexible enough to handle these challenges, the industry should evangelize the market accordingly.

A Finextra member
A Finextra member 09 March, 2015, 09:53Be the first to give this comment the thumbs up 0 likes

Perhaps not so much a question of speed but more of Agility. For example we have customers using packaged solutions like Calypso which are great, but adding features/upgrading takes a long time. If companies can streamline dev/test by adopting agile or DevOps practises, then the delivery becomes faster and quality (if done correctly) goes up. Delphix takes care of the data refresh, rest and rewind, but you need the rest to achieve real speed.

Anthony Walton
Anthony Walton - Iliad Solutions - Leeds 09 March, 2015, 12:47Be the first to give this comment the thumbs up 0 likes

It’s almost the perfect storm for introducing bugs into Payments Systems.


A clash of old and new technology

The increasing adoption of “Services” , often external

Agile and DevOps redefining what a delivery is, and how long you have to test it

The need to prototype, on-board and deliver more and more quicker and quicker.


It’s great to automate but this requires vendors to provide genuine end-to-end coverage, and not in 5 tools that don’t allow you to make changes yourself.


A Finextra member
A Finextra member 09 March, 2015, 12:511 like 1 like

@Anthony - I have to agree with you on the need for end-to-end coverage and ideally with an integrated test harness that validates end-to-end and not just at each end

Now hiring