28 June 2017
download the report now

Australian banks suffer ATM problems after cloud service outage

06 June 2016  |  8382 views  |  18 Cloud

Millions of Australian banking customers were unable to access ATM and Eftpos services on Sunday after severe storms on the east coast caused widespread damage, including a major outage at a cloud computing service used by several banks.

Amazon Web Services, a web hosting platform popular with several banks and retailers, has been blamed for many of the ATM and Eftpos problems which have affected a number of banks and their customers, including ME Bank and Commonwealth Bank. 

In addition payment providers such as First Data have also been blamed by some of the banks. 

Unable to use their bank cards, shoppers had to abandon their trolleys at supermarkets and frustration soon spilled over onto social media.

As of Monday, most faults have been fixed. Bank of Queensland reported that its ATM and Eftpos services had been restored late on Sunday evening. 

Meanwhile, in a statement to Finextra, Westpac says: "As of Monday morning Westpac Group mobile banking services were restored after an outage late Sunday. While a small proportion of customers experienced issues with Westpac ATM and EFTPOS services late Sunday, most customers were unaffected. This issue was not connected to the Amazon Web Services (AWS) outage.

"While we aim to ensure continuity of our systems, the severe storm system created disruptions across our network which impacted our services. We sincerely apologise to our customers for any inconvenience."

These latest problems come less than a month after four banks - Westpac, St George, BankSA and Bank of Melbourne - reported problems with their ATM networks. The repeated failures have raised the issue of banks' reliance on third party providers and data centres and their failure to backup their servers at alternative locations and thus avoid the problems caused by localised storms.

Comments: (18)

Jean Kaady
Jean Kaady - MySmartContact.com - Beirut | 06 June, 2016, 11:35

So Amazon cloud should relocate to "stormless" sites?

You see concentration in one place is always the problem. 

Look at successful examples- Nature: Dispersed fauna & flora. Cellular: Spread of connectivity across areas. 

Outsourcing to clouds does mostly make sense for businesses, but who said clouds shouldn't be thinly & smartly spread ?

1 thumb up! 1 thumb up! (Log in to thumb up)
Matt Scott
Matt Scott - RenovITe Technologies Inc - London | 06 June, 2016, 11:41 Sounds like a lack of Comms resilience - always build network seperacy in bu design. I have seen too many Banks and Mega Merchants hit major outages due to poor comms network design.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Matt Scott
Matt Scott - RenovITe Technologies Inc - London | 06 June, 2016, 11:42 *by design. I blame the iOS keyboard and not my fat fingers - honest
1 thumb up! 1 thumb up! (Log in to thumb up)
David Birch
David Birch - Tomorrow's Transactions - London | 06 June, 2016, 15:51

Say 20m bank accounts, say a few hundred Kb for each to record last N transactions, a 16Gb memory card could store the whole ledger. Put debit card system on the blockchain so there's nothing in the middle to go down anymore.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 06 June, 2016, 19:50

This won't stop the digerati from blaming legacy the next time RBS' payment networks go down! On another note, I should add this as a sure-fire way to kill cash.

How To Really Kill Cash 

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Matt Scott
Matt Scott - RenovITe Technologies Inc - London | 06 June, 2016, 19:55 BlockChain 2.0, the sequel. The Consultants are back and this time they're invoicing... Not convinced on the USB idea - I think they used to do something similar on reel to reel tape. I'd much rather use a distributed data aggregation service to ensure the BlockChain ledger survived the apocolypse unscathed.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 08:21

So... "a storm caused a major outage at a cloud computing service"?? Surely these banks did not have everything in a single data processing center. Or did they? I can't believe they haven't heard of, or implemented, geo-replication for a critical service like this. If they didn't have replication implemented, blaming the cloud service vendor seems short-sighted.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
A Finextra member
A Finextra member | 07 June, 2016, 08:45

Having alternate facilities to shift the workload to in case of a site outage is quite helpful, but it is also very helpful to have the required data already sitting there. There have been cases (eg. the recent Salesforce.com outage in the US) where apparently the data had still to be moved over to the alternate site - which can take a few hours. However, taking such precautions will result in additional cost ...

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 09:18

@Gerhard... I fully agree with you that data needs to be already available at the alternate site. I should have probably been more clear and explained I referred to real-time / on-line geo-replication. In my ignorance I don't know if that is supported by AWS, however we use it in our Azure services... and we're not a bank processing critical client transactions and services...! The cost overhead is actually very low as most of the infrastructure in the secondary site does not need to be spun-up until actually needed.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 10:24

Banks have known about fault-tolerant, clustering, ACTIVE:ACTIVE and other onprem redundancy / failover technologies and, where justified, implemented them ages ago. I remember a midsized Indian bank buying a Tandem mainframe with 100% fault tolerance to run its ATM switch 25 years ago. I doubt if the banking industry needs guidance on how to go about securing its infrastructure.

Point is, after migrating to the cloud, why the heck should banks still have to worry about infra? Every cloud service provider promises to shield end users from infra worries. High time PaaS / IaaS vendors fulfilled their promise or stopped making tall claims like this. Otherwise, whatever little legacy transformation is happening in BFSI will stop.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 10:39

@Ketharaman... I agree, but I believe a big mistake a lot of people make is that when you port Financial Services to "the cloud", the route being followed is highly secure, locked-down, environments that are IAAS... where the customer - not the cloud vendor - still has to define and manage their redundancy needs. There are significant economic benefits to this still. But I'm not sure it's fair to lay the reponsibility for Business Continuity wholely on the doorstep of the cloud vendor.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 11:16

@DanielSmith: If these banks were to sue AWS, Amazon would win the case hands down - I'm sure there's adequate fineprint in an AWS contract to put the blame back on the banks. But then, what's the difference between these purportedly CX-driven new-age companies and the purportedly-price-gouging-hidden-cost-laden old-age banks?

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 11:43

@Ketharaman... :-) It's really just a question of perceived flexibility and economics. The fact that you / we can spin-up additional resources very quickly as and when needed, and take the same down again when the workload drops off... all without the traditional Capex.

But - and I think we agree - there's a gap in people's perceptions about who really needs to take responsibility for ensuring business continuity in this virtual world.

Today, in my mind, it is still the customer, and has to be. Tomorrow, maybe a cloud vendor comes up with a solution that allows them to handle the same without creating any data / access security concerns. If they do, and Azure is maybe closer than AWS to this with their PAAS offerings... then they will bring huge value to the table.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 13:17

@DanielSmith: I agree with you about the crux of the offering. To take an analogy, if my factory suffers a power failure, the only way I can ensure business continuity is by having a DG set that will provide standby power. My argument is about messaging and communications: My utility company upfront tells me to invest in a DG set if I'm expecting to run my product facility 24/7/365. Whereas what I hear from any PaaS / IaaS provider is "leave your infrastructure worries to us". As a technology marketer, I shouldn't complain but, as a technology consumer, I can't help doing so!Therein lies the disconnect:)

1 thumb up! 1 thumb up! (Log in to thumb up)
Matt Scott
Matt Scott - RenovITe Technologies Inc - London | 07 June, 2016, 14:13 Before we jump to the conclusion that it is the Cloud/PaaS environments fault - have we considered that perhaps the Applications were not implemented correctly by the customers? Migrating into a Cloud/PaaS environment has serious implications and systems often need significant redesign to operate as expected in such environments - and enabling leverage of features such as demand elasticity. I still think this is a sloppy comms design issue.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Tom Howarth
Tom Howarth - CapGemini - Preston | 06 September, 2016, 09:59

A little bit late to the party here, but Matt makes a valid point, we are all too quick to point the finger.  we do know have any real knowledge of what services/applcation were ported to the public Cloud, we have no knowledge of where the failure occured.  All we know that it affected ATM and ETFPOS devices in some cases and mobile banking on others.  from my experiance these three services do not share any commonality external to the banks systems, but they do interact with and travel over many third party services.  Some services are more easily removed from study, than others.

Root Cause analysis of a fault should not be left to the masses.  it may come to fruition that AWS is to fault, but that does not mean that AWS is to blame, at the end of the day a company is still responsible for their Data and it's access, this cannot be offloaded ot a provider.  Resilance and redundancy are different sides of the same coin and putting a front end customer facing system, coupled with a main method of moving money into a inherently none redundant system is foolish,  AWS's idea of a failure domain is significantly different to what the average layman not working on global Cloud infrastructure would consider a failure domain.  it is very likely that AWS are not even in breach of any SLA's regarding this outage.

To me the old addage of Buyer beware is so apt with Public Cloud services.  if am person comes with a deal that looks too good to be true. it very likely is.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
A Finextra member
A Finextra member | 06 September, 2016, 10:48

To me, it all boils down to the type of infrastructure the bank or other customer organization chooses to run their application on. For instance, they can decide for running on a very reliable and secure infrastructure as described by Ketharaman further up in this discussion string. However, if they want to run in the cloud and still want the same kind of reliability and security, they need to make sure that their cloud provider uses that same kind of infrastructure. Failing to do so and then trying to blame the cloud provider just makes no sense.

AWS et al do not provide such reliable and secure infrastructure at this time. However, there are payments service providers using such infrastructure and offering their service to banks since decades, long before the term "SaaS" had been invented. A good share of the world's payments are run by such service providers on behalf of banks who do not want to run such systems themselves, and this approach works nicely since many years.

However, assuming that general purpose cloud providers running on plain vanilla gear can deliver the same kind of infrastructure reliability at much lower cost just isn't very realistic. And of course, they can't deliver payments as SaaS either, thus a lot of effort and cost remains with the customer organization. 

2 thumb ups! 2 thumb ups! (Log in to thumb up)
Tom Howarth
Tom Howarth - CapGemini - Preston | 06 September, 2016, 10:58 I agree completely, my personal view is that Banks are expecting that the likes of AWS and Azure provide same level of service and resilience provided by "legacy" 3rd party payment service provided
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Comment on this story (membership required)

Finextra news in your inbox

For Finextra's free daily newsletter, breaking news flashes and weekly jobs board: sign up now

Related stories

Barclays Business Banking breaks down over long weekend

Barclays Business Banking breaks down over long weekend

03 May 2016  |  7163 views  |  0 comments | 3 tweets | 2 linkedin
CommBank customers rage on Twitter over 18-hour outage; NatWest clients duped by bogus account

CommBank customers rage on Twitter over 18-hour outage; NatWest clients duped by bogus account

11 September 2015  |  7430 views  |  0 comments | 6 tweets | 3 linkedin
Westpac to shrink bank branch, grow spending on digital

Westpac to shrink bank branch, grow spending on digital

08 September 2015  |  9508 views  |  0 comments | 18 tweets | 10 linkedin
NatWest blames online outage on DDoS attack

NatWest blames online outage on DDoS attack

31 July 2015  |  11183 views  |  0 comments | 16 tweets | 5 linkedin
Chaos in the City as Bloomberg terminals go down

Chaos in the City as Bloomberg terminals go down

17 April 2015  |  7596 views  |  0 comments | 13 tweets | 6 linkedin
CommBank debuts cardless ATM transactions

CommBank debuts cardless ATM transactions

30 April 2014  |  7454 views  |  1 comments | 4 tweets | 11 linkedin
Bats exchange hit by outage

Bats exchange hit by outage

07 August 2013  |  4606 views  |  0 comments | 1 tweets
Westpac hit by major online outage

Westpac hit by major online outage

01 August 2013  |  6568 views  |  0 comments | 2 tweets | 3 linkedin
NAB suffers five-hour outage

NAB suffers five-hour outage

05 September 2012  |  8855 views  |  0 comments
CBA customers left fuming by more online, ATM and eftpos outages

CBA customers left fuming by more online, ATM and eftpos outages

14 December 2011  |  9903 views  |  0 comments

Related company news

 

Related blogs

Create a blog about this story (membership required)
visit www.response.ncr.comvisit www.finastra.comvisit www.events.sap.com

Top topics

Most viewed Most shared
Live: EBAday 2017, day twoLive: EBAday 2017, day two
11415 views comments | 4 tweets | 5 linkedin
ECB preps eurozone-wide instant payments serviceECB preps eurozone-wide instant payments s...
8903 views comments | 19 tweets | 34 linkedin
ABN Amro tests wearable tech for contactless paymentsABN Amro tests wearable tech for contactle...
8402 views comments | 9 tweets | 6 linkedin
hands typing furiouslyArtificial Intelligence- Computer to IOT
8337 views 0 | 4 tweets | 2 linkedin

Featured job

Six Figure Base + Commission + Stock Options
London

Find your next job