Visa says 5.2m payments failed during 10 hour outage

Visa says 5.2m payments failed during 10 hour outage

More than five million transactions across Europe failed during Visa's 10 hour outage earlier this month, but the company's response has satisfied UK politicians probing the incident.

In a letter responding to questions from the UK's Treasury Select committee chair Nicky Morgan, Visa's European boss, Charlotte Hogg, says that 10% of 51.2 million transactions across Europe were affected during the outage, which lasted from 14.35 on 1 June until 00.45 the next day.

In the UK, 2.4 million transactions failed to process properly, with 1.7 million credit and debit cards affected.

Explaining what went wrong, Hogg says that a component in Visa's primary data centre "suffered a very rare partial failure". This meant that Visa's secondary data centre was not able to kick in properly and automatically process all transactions.

Explains Hogg: "As a result, it took far longer than expected to isolate the primary site; in the interim, the malfunctioning system at the primary data centre continued to try to synchronise messages with the secondary site. This created a backlog of messages at the secondary data centre, which, in turn, slowed down that site's ability to process incoming transactions."

Hogg also stresses that no money will leave cardholder accounts for transactions that were not completed, adding that Visa is working with UK banks to "address the impacts of the incident, including appropriate compensation of cardholders".

While technical steps have been taken to reduce the chances of a repeat failure, in the longer term Visa says that by the end of the year it will migrate European processing to its global system, VisaNet, which is "more resilient in its ability to detect and recover from partial malfunctions of the type that impacted our European system".

Visa looks set to escape further action from UK politicians, with Morgan replying to Hoggs letter: "The Treasury Committee is satisfied with Visa’s answers regarding its system failure earlier this month, which lasted just over 10 hours and saw 2.4 million transactions in the UK fail to process. It appears that the problems have been fully resolved.

"The news that debit card payments have overtaken cash use for the first time shows that the reliability of IT systems is becoming ever-more important. The detriment caused to consumers by IT failures is greater than ever, so the Committee will become less tolerant of them.

"The Committee expects to see the findings of the independent review, which will examine the lessons to be learned from the incident, in full."

Comments: (3)

A Finextra member
A Finextra member 19 June, 2018, 16:222 likes 2 likes

Visa is working with UK banks to "address the impacts of the incident, including appropriate compensation of cardholders"..... and what about compensating the merchants who are just as dependent on the Visa network and who  unlike consumers, are directly paying for the service? I bet they won't receive a penny, despite the lost sales, lost produce (as frozen food defrosts), increased resource to put everything back on the shelves as trolleys are abandoned and having to deal at the front line with irate customers! 

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 19 June, 2018, 19:08Be the first to give this comment the thumbs up 0 likes

The inverter battery backup at my home kicks in as soon as mains power fails (due to a fault in the line or planned load shedding). But it does not come on when there's a low voltage in the main supply. I was reminded of this when I read that Visa's secondary data center failed to kick in because of a partial failure of the primary system. 

Bob Lyddon
Bob Lyddon - Lyddon Consulting Services - Thames Ditton 28 June, 2018, 10:38Be the first to give this comment the thumbs up 0 likes

Have been away for a while so looking at this with fresh eyes. Is this a satisfactory explanation? For a critical national system? "It fell over because something went wrong that we didn't expect". That might satisfy Nicky Morgan, expectations of whom need to be set at a suitably subterranean level, but does it satisfy the trade?