Australian banks suffer ATM problems after cloud service outage

Australian banks suffer ATM problems after cloud service outage

Millions of Australian banking customers were unable to access ATM and Eftpos services on Sunday after severe storms on the east coast caused widespread damage, including a major outage at a cloud computing service used by several banks.

Amazon Web Services, a web hosting platform popular with several banks and retailers, has been blamed for many of the ATM and Eftpos problems which have affected a number of banks and their customers, including ME Bank and Commonwealth Bank. 

In addition payment providers such as First Data have also been blamed by some of the banks. 

Unable to use their bank cards, shoppers had to abandon their trolleys at supermarkets and frustration soon spilled over onto social media.

As of Monday, most faults have been fixed. Bank of Queensland reported that its ATM and Eftpos services had been restored late on Sunday evening. 

Meanwhile, in a statement to Finextra, Westpac says: "As of Monday morning Westpac Group mobile banking services were restored after an outage late Sunday. While a small proportion of customers experienced issues with Westpac ATM and EFTPOS services late Sunday, most customers were unaffected. This issue was not connected to the Amazon Web Services (AWS) outage.

"While we aim to ensure continuity of our systems, the severe storm system created disruptions across our network which impacted our services. We sincerely apologise to our customers for any inconvenience."

These latest problems come less than a month after four banks - Westpac, St George, BankSA and Bank of Melbourne - reported problems with their ATM networks. The repeated failures have raised the issue of banks' reliance on third party providers and data centres and their failure to backup their servers at alternative locations and thus avoid the problems caused by localised storms.

Comments: (18)

A Finextra member
A Finextra member 06 June, 2016, 11:351 like 1 like

So Amazon cloud should relocate to "stormless" sites?

You see concentration in one place is always the problem. 

Look at successful examples- Nature: Dispersed fauna & flora. Cellular: Spread of connectivity across areas. 

Outsourcing to clouds does mostly make sense for businesses, but who said clouds shouldn't be thinly & smartly spread ?

Matt Scott
Matt Scott - RenovITe Technologies Inc - London 06 June, 2016, 11:41Be the first to give this comment the thumbs up 0 likes Sounds like a lack of Comms resilience - always build network seperacy in bu design. I have seen too many Banks and Mega Merchants hit major outages due to poor comms network design.
Matt Scott
Matt Scott - RenovITe Technologies Inc - London 06 June, 2016, 11:421 like 1 like *by design. I blame the iOS keyboard and not my fat fingers - honest
David Birch
David Birch - Tomorrow's Transactions - London 06 June, 2016, 15:51Be the first to give this comment the thumbs up 0 likes

Say 20m bank accounts, say a few hundred Kb for each to record last N transactions, a 16Gb memory card could store the whole ledger. Put debit card system on the blockchain so there's nothing in the middle to go down anymore.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 06 June, 2016, 19:50Be the first to give this comment the thumbs up 0 likes

This won't stop the digerati from blaming legacy the next time RBS' payment networks go down! On another note, I should add this as a sure-fire way to kill cash.

How To Really Kill Cash 

Matt Scott
Matt Scott - RenovITe Technologies Inc - London 06 June, 2016, 19:55Be the first to give this comment the thumbs up 0 likes BlockChain 2.0, the sequel. The Consultants are back and this time they're invoicing... Not convinced on the USB idea - I think they used to do something similar on reel to reel tape. I'd much rather use a distributed data aggregation service to ensure the BlockChain ledger survived the apocolypse unscathed.
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York 07 June, 2016, 08:21Be the first to give this comment the thumbs up 0 likes

So... "a storm caused a major outage at a cloud computing service"?? Surely these banks did not have everything in a single data processing center. Or did they? I can't believe they haven't heard of, or implemented, geo-replication for a critical service like this. If they didn't have replication implemented, blaming the cloud service vendor seems short-sighted.

A Finextra member
A Finextra member 07 June, 2016, 08:45Be the first to give this comment the thumbs up 0 likes

Having alternate facilities to shift the workload to in case of a site outage is quite helpful, but it is also very helpful to have the required data already sitting there. There have been cases (eg. the recent Salesforce.com outage in the US) where apparently the data had still to be moved over to the alternate site - which can take a few hours. However, taking such precautions will result in additional cost ...

Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York 07 June, 2016, 09:18Be the first to give this comment the thumbs up 0 likes

@Gerhard... I fully agree with you that data needs to be already available at the alternate site. I should have probably been more clear and explained I referred to real-time / on-line geo-replication. In my ignorance I don't know if that is supported by AWS, however we use it in our Azure services... and we're not a bank processing critical client transactions and services...! The cost overhead is actually very low as most of the infrastructure in the secondary site does not need to be spun-up until actually needed.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 07 June, 2016, 10:24Be the first to give this comment the thumbs up 0 likes

Banks have known about fault-tolerant, clustering, ACTIVE:ACTIVE and other onprem redundancy / failover technologies and, where justified, implemented them ages ago. I remember a midsized Indian bank buying a Tandem mainframe with 100% fault tolerance to run its ATM switch 25 years ago. I doubt if the banking industry needs guidance on how to go about securing its infrastructure.

Point is, after migrating to the cloud, why the heck should banks still have to worry about infra? Every cloud service provider promises to shield end users from infra worries. High time PaaS / IaaS vendors fulfilled their promise or stopped making tall claims like this. Otherwise, whatever little legacy transformation is happening in BFSI will stop.

Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York 07 June, 2016, 10:39Be the first to give this comment the thumbs up 0 likes

@Ketharaman... I agree, but I believe a big mistake a lot of people make is that when you port Financial Services to "the cloud", the route being followed is highly secure, locked-down, environments that are IAAS... where the customer - not the cloud vendor - still has to define and manage their redundancy needs. There are significant economic benefits to this still. But I'm not sure it's fair to lay the reponsibility for Business Continuity wholely on the doorstep of the cloud vendor.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 07 June, 2016, 11:16Be the first to give this comment the thumbs up 0 likes

@DanielSmith: If these banks were to sue AWS, Amazon would win the case hands down - I'm sure there's adequate fineprint in an AWS contract to put the blame back on the banks. But then, what's the difference between these purportedly CX-driven new-age companies and the purportedly-price-gouging-hidden-cost-laden old-age banks?

Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York 07 June, 2016, 11:43Be the first to give this comment the thumbs up 0 likes

@Ketharaman... :-) It's really just a question of perceived flexibility and economics. The fact that you / we can spin-up additional resources very quickly as and when needed, and take the same down again when the workload drops off... all without the traditional Capex.

But - and I think we agree - there's a gap in people's perceptions about who really needs to take responsibility for ensuring business continuity in this virtual world.

Today, in my mind, it is still the customer, and has to be. Tomorrow, maybe a cloud vendor comes up with a solution that allows them to handle the same without creating any data / access security concerns. If they do, and Azure is maybe closer than AWS to this with their PAAS offerings... then they will bring huge value to the table.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 07 June, 2016, 13:171 like 1 like

@DanielSmith: I agree with you about the crux of the offering. To take an analogy, if my factory suffers a power failure, the only way I can ensure business continuity is by having a DG set that will provide standby power. My argument is about messaging and communications: My utility company upfront tells me to invest in a DG set if I'm expecting to run my product facility 24/7/365. Whereas what I hear from any PaaS / IaaS provider is "leave your infrastructure worries to us". As a technology marketer, I shouldn't complain but, as a technology consumer, I can't help doing so!Therein lies the disconnect:)

Matt Scott
Matt Scott - RenovITe Technologies Inc - London 07 June, 2016, 14:13Be the first to give this comment the thumbs up 0 likes Before we jump to the conclusion that it is the Cloud/PaaS environments fault - have we considered that perhaps the Applications were not implemented correctly by the customers? Migrating into a Cloud/PaaS environment has serious implications and systems often need significant redesign to operate as expected in such environments - and enabling leverage of features such as demand elasticity. I still think this is a sloppy comms design issue.
A Finextra member
A Finextra member 06 September, 2016, 09:59Be the first to give this comment the thumbs up 0 likes

A little bit late to the party here, but Matt makes a valid point, we are all too quick to point the finger.  we do know have any real knowledge of what services/applcation were ported to the public Cloud, we have no knowledge of where the failure occured.  All we know that it affected ATM and ETFPOS devices in some cases and mobile banking on others.  from my experiance these three services do not share any commonality external to the banks systems, but they do interact with and travel over many third party services.  Some services are more easily removed from study, than others.

Root Cause analysis of a fault should not be left to the masses.  it may come to fruition that AWS is to fault, but that does not mean that AWS is to blame, at the end of the day a company is still responsible for their Data and it's access, this cannot be offloaded ot a provider.  Resilance and redundancy are different sides of the same coin and putting a front end customer facing system, coupled with a main method of moving money into a inherently none redundant system is foolish,  AWS's idea of a failure domain is significantly different to what the average layman not working on global Cloud infrastructure would consider a failure domain.  it is very likely that AWS are not even in breach of any SLA's regarding this outage.

To me the old addage of Buyer beware is so apt with Public Cloud services.  if am person comes with a deal that looks too good to be true. it very likely is.

A Finextra member
A Finextra member 06 September, 2016, 10:481 like 1 like

To me, it all boils down to the type of infrastructure the bank or other customer organization chooses to run their application on. For instance, they can decide for running on a very reliable and secure infrastructure as described by Ketharaman further up in this discussion string. However, if they want to run in the cloud and still want the same kind of reliability and security, they need to make sure that their cloud provider uses that same kind of infrastructure. Failing to do so and then trying to blame the cloud provider just makes no sense.

AWS et al do not provide such reliable and secure infrastructure at this time. However, there are payments service providers using such infrastructure and offering their service to banks since decades, long before the term "SaaS" had been invented. A good share of the world's payments are run by such service providers on behalf of banks who do not want to run such systems themselves, and this approach works nicely since many years.

However, assuming that general purpose cloud providers running on plain vanilla gear can deliver the same kind of infrastructure reliability at much lower cost just isn't very realistic. And of course, they can't deliver payments as SaaS either, thus a lot of effort and cost remains with the customer organization. 

A Finextra member
A Finextra member 06 September, 2016, 10:58Be the first to give this comment the thumbs up 0 likes I agree completely, my personal view is that Banks are expecting that the likes of AWS and Azure provide same level of service and resilience provided by "legacy" 3rd party payment service provided