21 August 2017
visit www.avoka.com

Anatomy of a systems failure: Danske Bank turns spotlight on DB2

03 April 2003  |  5855 views  |  0 pda - old

Danske Bank has issued a statement outlining how a series of never-before-identified errors in IBM's DB2 software caused a week-long disruption to payments systems and the trading and settlement of currencies and securities for customers.

The problems stemmed from a routine replacement by IBM engineers at a Danske Bank operations centre of a defective electrical unit in an RVA disk system used for storage of data in the DB2 database. During the repair process, there was an electrical outage in the disk system which sparked a critical operational situation with wide-ranging effects, says the Danish bank.

The RVA disk and the business data stored on it were not accessible to the bank's core processing systems after the breakdown. This meant that the part of the systems running at the bank's operating centre in Ejby (currency and securities trading, as well as foreign systems and payments) was not able to function. Operations continued for the systems running at the operating centre in Brabrand which runs the bank's cash dispensers and self-service systems.

When the problematic RVA disk unit was given the all-clear later that day, the bank's systems in Ejby were restarted as usual. However, by early Tuesday morning it had become apparent that the first batch runs out of Ejby contained data inconsistencies.

"This first software error in DB2 database software had existed in all similar installations since 1997, without IBM's knowledge," says the bank.

In the following process of data recovery, which lasted from the morning of Tuesday, 11 March, until Friday, 14 March, three more errors were discovered in the software, affecting DB2 tables and recovery jobs and resulting in new episodes of inconsistent data.

In order to avoid longer delays, Danske decided not to wait for an IBM patch, but instead rebuilt the database using back-up data stored at the operating centre in Drarbrand, which itself was beginning to suffer from data deficiencies at Ejby.

A week after the inital fault was discovered, all data from the problematic disk was transferred to other systems, and the disk was taken out of operation and the bank settled all accumulated transactions with external counterparites.

"None of the errors in DB2 were previously known to IBM, and therefore no patches could immediately be provided," says the bank.

Patches are now available for all users of DB2 database software and have since been installed by Danske, which is additionally seeking compensation from the supplier for the problems caused.

The bank says it is also discussing with IBM measures to improve DB2 as well as more extensive and efficient emergency procedures in case of future critical events.

Internally, the bank is considering considering future investment in its two-centre operations to protect some of the group's vital IT systems, but it refutes press reports that the implementation of mirrored disks at the centres would have prevented the failures.

Responding to customer feedback for more open lines of communication in future emergencies, Danske says it has rewritten its contingency plans to address these concerns.

Comments: (0)

Comment on this story (membership required)

Finextra news in your inbox

For Finextra's free daily newsletter, breaking news flashes and weekly jobs board: sign up now

Related stories

CLS glitch hits Asia Pacific trades

CLS glitch hits Asia Pacific trades

27 March 2003  |  4027 views  |  0 comments
Danske Bank to compensate customers after systems break down

Danske Bank to compensate customers after systems break down

14 March 2003  |  3942 views  |  0 comments
UK Treasury probes financial market disaster planning

UK Treasury probes financial market disaster planning

25 February 2003  |  3525 views  |  0 comments
GAO slams regulators and financial firms for disaster recovery failings

GAO slams regulators and financial firms for disaster recovery failings

13 February 2003  |  4625 views  |  0 comments
OM software blamed for Hong Kong exchange failure

OM software blamed for Hong Kong exchange failure

06 February 2003  |  7754 views  |  0 comments

Related company news

 

Related blogs

Create a blog about this story (membership required)
visit www.niceactimize.comvisit www.abe-eba.euvisit www.worldpaymentsreport.com

Who is commenting?

A Finextra member Finextra Member Commented on: Real-time payments in...
A Finextra member Finextra Member Commented on: Barclays uses sensors...

Top topics

Most viewed Most shared
Mobile contactless spending accelerating in UKMobile contactless spending accelerating i...
11603 views comments | 26 tweets | 23 linkedin
Barclays pairs banking data with third party apps for SmartBusiness DashboardBarclays pairs banking data with third par...
9901 views comments | 22 tweets | 31 linkedin
hands typing furiouslyWhy Is Risk Analytics Important?
9693 views 0 | 5 tweets | 1 linkedin
Norwegian banks and startups form fintech clusterNorwegian banks and startups form fintech...
9399 views comments | 19 tweets | 23 linkedin
RBS to bring Silicon Valley to EdinburghRBS to bring Silicon Valley to Edinburgh
9365 views comments | 10 tweets | 8 linkedin

Featured job

Competitive
New York, NY - USA (some flexibility on location)

Find your next job