25 March 2017
visit nextgenbanking.co.uk

Australian banks suffer ATM problems after cloud service outage

06 June 2016  |  7902 views  |  18 Cloud

Millions of Australian banking customers were unable to access ATM and Eftpos services on Sunday after severe storms on the east coast caused widespread damage, including a major outage at a cloud computing service used by several banks.

Amazon Web Services, a web hosting platform popular with several banks and retailers, has been blamed for many of the ATM and Eftpos problems which have affected a number of banks and their customers, including ME Bank and Commonwealth Bank. 

In addition payment providers such as First Data have also been blamed by some of the banks. 

Unable to use their bank cards, shoppers had to abandon their trolleys at supermarkets and frustration soon spilled over onto social media.

As of Monday, most faults have been fixed. Bank of Queensland reported that its ATM and Eftpos services had been restored late on Sunday evening. 

Meanwhile, in a statement to Finextra, Westpac says: "As of Monday morning Westpac Group mobile banking services were restored after an outage late Sunday. While a small proportion of customers experienced issues with Westpac ATM and EFTPOS services late Sunday, most customers were unaffected. This issue was not connected to the Amazon Web Services (AWS) outage.

"While we aim to ensure continuity of our systems, the severe storm system created disruptions across our network which impacted our services. We sincerely apologise to our customers for any inconvenience."

These latest problems come less than a month after four banks - Westpac, St George, BankSA and Bank of Melbourne - reported problems with their ATM networks. The repeated failures have raised the issue of banks' reliance on third party providers and data centres and their failure to backup their servers at alternative locations and thus avoid the problems caused by localised storms.

Comments: (18)

Jean Kaady
Jean Kaady - Hey Pay - beirut | 06 June, 2016, 11:35

So Amazon cloud should relocate to "stormless" sites?

You see concentration in one place is always the problem. 

Look at successful examples- Nature: Dispersed fauna & flora. Cellular: Spread of connectivity across areas. 

Outsourcing to clouds does mostly make sense for businesses, but who said clouds shouldn't be thinly & smartly spread ?

1 thumb up! 1 thumb up! (Log in to thumb up)
Matt Scott
Matt Scott - NCR Corporation - London | 06 June, 2016, 11:41 Sounds like a lack of Comms resilience - always build network seperacy in bu design. I have seen too many Banks and Mega Merchants hit major outages due to poor comms network design.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Matt Scott
Matt Scott - NCR Corporation - London | 06 June, 2016, 11:42 *by design. I blame the iOS keyboard and not my fat fingers - honest
1 thumb up! 1 thumb up! (Log in to thumb up)
David Birch
David Birch - Tomorrow's Transactions - London | 06 June, 2016, 15:51

Say 20m bank accounts, say a few hundred Kb for each to record last N transactions, a 16Gb memory card could store the whole ledger. Put debit card system on the blockchain so there's nothing in the middle to go down anymore.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 06 June, 2016, 19:50

This won't stop the digerati from blaming legacy the next time RBS' payment networks go down! On another note, I should add this as a sure-fire way to kill cash.

How To Really Kill Cash 

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Matt Scott
Matt Scott - NCR Corporation - London | 06 June, 2016, 19:55 BlockChain 2.0, the sequel. The Consultants are back and this time they're invoicing... Not convinced on the USB idea - I think they used to do something similar on reel to reel tape. I'd much rather use a distributed data aggregation service to ensure the BlockChain ledger survived the apocolypse unscathed.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 08:21

So... "a storm caused a major outage at a cloud computing service"?? Surely these banks did not have everything in a single data processing center. Or did they? I can't believe they haven't heard of, or implemented, geo-replication for a critical service like this. If they didn't have replication implemented, blaming the cloud service vendor seems short-sighted.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Gerhard Schwartz
Gerhard Schwartz - Hewlett-Packard - | 07 June, 2016, 08:45

Having alternate facilities to shift the workload to in case of a site outage is quite helpful, but it is also very helpful to have the required data already sitting there. There have been cases (eg. the recent Salesforce.com outage in the US) where apparently the data had still to be moved over to the alternate site - which can take a few hours. However, taking such precautions will result in additional cost ...

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 09:18

@Gerhard... I fully agree with you that data needs to be already available at the alternate site. I should have probably been more clear and explained I referred to real-time / on-line geo-replication. In my ignorance I don't know if that is supported by AWS, however we use it in our Azure services... and we're not a bank processing critical client transactions and services...! The cost overhead is actually very low as most of the infrastructure in the secondary site does not need to be spun-up until actually needed.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 10:24

Banks have known about fault-tolerant, clustering, ACTIVE:ACTIVE and other onprem redundancy / failover technologies and, where justified, implemented them ages ago. I remember a midsized Indian bank buying a Tandem mainframe with 100% fault tolerance to run its ATM switch 25 years ago. I doubt if the banking industry needs guidance on how to go about securing its infrastructure.

Point is, after migrating to the cloud, why the heck should banks still have to worry about infra? Every cloud service provider promises to shield end users from infra worries. High time PaaS / IaaS vendors fulfilled their promise or stopped making tall claims like this. Otherwise, whatever little legacy transformation is happening in BFSI will stop.

1 thumb up! 1 thumb up! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 10:39

@Ketharaman... I agree, but I believe a big mistake a lot of people make is that when you port Financial Services to "the cloud", the route being followed is highly secure, locked-down, environments that are IAAS... where the customer - not the cloud vendor - still has to define and manage their redundancy needs. There are significant economic benefits to this still. But I'm not sure it's fair to lay the reponsibility for Business Continuity wholely on the doorstep of the cloud vendor.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 11:16

@DanielSmith: If these banks were to sue AWS, Amazon would win the case hands down - I'm sure there's adequate fineprint in an AWS contract to put the blame back on the banks. But then, what's the difference between these purportedly CX-driven new-age companies and the purportedly-price-gouging-hidden-cost-laden old-age banks?

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Daniel Smith
Daniel Smith - CHOICE Financial Solutions - New York | 07 June, 2016, 11:43

@Ketharaman... :-) It's really just a question of perceived flexibility and economics. The fact that you / we can spin-up additional resources very quickly as and when needed, and take the same down again when the workload drops off... all without the traditional Capex.

But - and I think we agree - there's a gap in people's perceptions about who really needs to take responsibility for ensuring business continuity in this virtual world.

Today, in my mind, it is still the customer, and has to be. Tomorrow, maybe a cloud vendor comes up with a solution that allows them to handle the same without creating any data / access security concerns. If they do, and Azure is maybe closer than AWS to this with their PAAS offerings... then they will bring huge value to the table.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 07 June, 2016, 13:17

@DanielSmith: I agree with you about the crux of the offering. To take an analogy, if my factory suffers a power failure, the only way I can ensure business continuity is by having a DG set that will provide standby power. My argument is about messaging and communications: My utility company upfront tells me to invest in a DG set if I'm expecting to run my product facility 24/7/365. Whereas what I hear from any PaaS / IaaS provider is "leave your infrastructure worries to us". As a technology marketer, I shouldn't complain but, as a technology consumer, I can't help doing so!Therein lies the disconnect:)

2 thumb ups! 2 thumb ups! (Log in to thumb up)
Matt Scott
Matt Scott - NCR Corporation - London | 07 June, 2016, 14:13 Before we jump to the conclusion that it is the Cloud/PaaS environments fault - have we considered that perhaps the Applications were not implemented correctly by the customers? Migrating into a Cloud/PaaS environment has serious implications and systems often need significant redesign to operate as expected in such environments - and enabling leverage of features such as demand elasticity. I still think this is a sloppy comms design issue.
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Tom Howarth
Tom Howarth - CapGemini - Preston | 06 September, 2016, 09:59

A little bit late to the party here, but Matt makes a valid point, we are all too quick to point the finger.  we do know have any real knowledge of what services/applcation were ported to the public Cloud, we have no knowledge of where the failure occured.  All we know that it affected ATM and ETFPOS devices in some cases and mobile banking on others.  from my experiance these three services do not share any commonality external to the banks systems, but they do interact with and travel over many third party services.  Some services are more easily removed from study, than others.

Root Cause analysis of a fault should not be left to the masses.  it may come to fruition that AWS is to fault, but that does not mean that AWS is to blame, at the end of the day a company is still responsible for their Data and it's access, this cannot be offloaded ot a provider.  Resilance and redundancy are different sides of the same coin and putting a front end customer facing system, coupled with a main method of moving money into a inherently none redundant system is foolish,  AWS's idea of a failure domain is significantly different to what the average layman not working on global Cloud infrastructure would consider a failure domain.  it is very likely that AWS are not even in breach of any SLA's regarding this outage.

To me the old addage of Buyer beware is so apt with Public Cloud services.  if am person comes with a deal that looks too good to be true. it very likely is.

1 thumb up! 1 thumb up! (Log in to thumb up)
Gerhard Schwartz
Gerhard Schwartz - Hewlett-Packard - | 06 September, 2016, 10:48

To me, it all boils down to the type of infrastructure the bank or other customer organization chooses to run their application on. For instance, they can decide for running on a very reliable and secure infrastructure as described by Ketharaman further up in this discussion string. However, if they want to run in the cloud and still want the same kind of reliability and security, they need to make sure that their cloud provider uses that same kind of infrastructure. Failing to do so and then trying to blame the cloud provider just makes no sense.

AWS et al do not provide such reliable and secure infrastructure at this time. However, there are payments service providers using such infrastructure and offering their service to banks since decades, long before the term "SaaS" had been invented. A good share of the world's payments are run by such service providers on behalf of banks who do not want to run such systems themselves, and this approach works nicely since many years.

However, assuming that general purpose cloud providers running on plain vanilla gear can deliver the same kind of infrastructure reliability at much lower cost just isn't very realistic. And of course, they can't deliver payments as SaaS either, thus a lot of effort and cost remains with the customer organization. 

2 thumb ups! 2 thumb ups! (Log in to thumb up)
Tom Howarth
Tom Howarth - CapGemini - Preston | 06 September, 2016, 10:58 I agree completely, my personal view is that Banks are expecting that the likes of AWS and Azure provide same level of service and resilience provided by "legacy" 3rd party payment service provided
Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Comment on this story (membership required)

Finextra news in your inbox

For Finextra's free daily newsletter, breaking news flashes and weekly jobs board: sign up now

Related stories

Barclays Business Banking breaks down over long weekend

Barclays Business Banking breaks down over long weekend

03 May 2016  |  6767 views  |  0 comments | 3 tweets | 2 linkedin
CommBank customers rage on Twitter over 18-hour outage; NatWest clients duped by bogus account

CommBank customers rage on Twitter over 18-hour outage; NatWest clients duped by bogus account

11 September 2015  |  7174 views  |  0 comments | 6 tweets | 3 linkedin
Westpac to shrink bank branch, grow spending on digital

Westpac to shrink bank branch, grow spending on digital

08 September 2015  |  9286 views  |  0 comments | 18 tweets | 10 linkedin
NatWest blames online outage on DDoS attack

NatWest blames online outage on DDoS attack

31 July 2015  |  10881 views  |  0 comments | 16 tweets | 5 linkedin
Chaos in the City as Bloomberg terminals go down

Chaos in the City as Bloomberg terminals go down

17 April 2015  |  7327 views  |  0 comments | 13 tweets | 6 linkedin
CommBank debuts cardless ATM transactions

CommBank debuts cardless ATM transactions

30 April 2014  |  7320 views  |  1 comments | 4 tweets | 11 linkedin
Bats exchange hit by outage

Bats exchange hit by outage

07 August 2013  |  4539 views  |  0 comments | 1 tweets
Westpac hit by major online outage

Westpac hit by major online outage

01 August 2013  |  6468 views  |  0 comments | 2 tweets | 3 linkedin
NAB suffers five-hour outage

NAB suffers five-hour outage

05 September 2012  |  8742 views  |  0 comments
CBA customers left fuming by more online, ATM and eftpos outages

CBA customers left fuming by more online, ATM and eftpos outages

14 December 2011  |  9792 views  |  0 comments

Related company news

 

Related blogs

Create a blog about this story (membership required)
Visit capgemini.comvisit abe-eba.eu

Top topics

Most viewed Most shared
French retailer Carrefour launches online bank accountFrench retailer Carrefour launches online...
57164 views comments | 15 tweets | 35 linkedin
European Commission opens public consultation on fintechEuropean Commission opens public consultat...
8762 views comments | 42 tweets | 25 linkedin
Can banks really win in the payments business of the future? – new Finextra reportCan banks really win in the payments busin...
8575 views comments | 23 tweets | 37 linkedin
Westpac wants to take over your messaging keyboardWestpac wants to take over your messaging...
7100 views comments | 4 tweets | 10 linkedin
SecureKey taps IBM to put identity on the blockchainSecureKey taps IBM to put identity on the...
7037 views comments | 22 tweets | 15 linkedin

Featured job

Six Figure Base + Commission + Stock Options
London

Find your next job