RBS meltdown blamed on Indian IT operative

RBS meltdown blamed on Indian IT operative

Royal Bank of Scotland is refusing to confirm or deny reports that an inexperienced IT operative in India was partially responsible for the computer meltdown that afflicted payments processing at the bank last week.

According to reports in UK tech rag The Register, the junior technician in India accidentally wiped out a queue of batched payments while backing out of a routine software upgrade.

Quoting a 'source familiar with the matter', the Register says that the situation spiralled out of control during an attempt to reverse a planned upgrade to the bank's mainframe-hosted CA-7 workload automation engine.

"When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling," says the source. "That created a large backlog as all the wiped information had to be re-inputted to the system and reprocessed. A complicated legacy mainframe system at RBS and a team inexperienced in its quirks made the problem harder to fix."

Earlier this year the bank moved to recruit CA-7 batch support staff in Hyderabad on a salary of £9000-£11,000 per annum while slashing higher paid equivalents in the UK.

If substantiated, the revelations could prove damaging for the bank's embattled boss Stephen Hester, who on Tuesday denied claims that the breakdown was in any way connected to underinvestment in IT and the offshoring of jobs to India.

In an interview with Sky News that touched on the outsourcing rumours, Hester said: "Well I have no evidence of that. The IT centre - our main centre, we're standing outside here in Edinburgh, [is] nothing to do with overseas. Our UK backbone has seen substantial investment."

Analysts at Ovum point to the problems faced by large banks in running mission-critical processes on ageing legacy systems with a diminishing pool of skilled and experienced staff.

"With most banks under heavy cost pressures, this means relatively junior staff are often given responsibility for systems where they have little experience beyond the routine, particularly in a stress situation (as with RBS) where things go outside normal operations," states Ovum. "The challenge for banks is that an operation (e.g. the software update to CA-7) that would score relatively low on the risk register escalated rapidly into a critical error. It illustrates that conventional cost versus risk decisions can be problematic, particularly for mission-critical (and their supporting) systems."

Separately, the FT is reporting that RBS is considering legal action against Computer Associates, the supplier of the CA-7 software. The board of the bank spent much of Tuesday in crisis talks, says the FT, after which it decided to launch a full-scale review.

The FT quotes an insider at the meeting: "It was certainly an issue with this software. We will still have to establish if this was their fault or if it was our handling of the software."

These are choppy waters for the bailed-out bank group, with politicians also demanding answers. The chairman of the Treasury Select Committee, Andrew Tyrie MP, has written to RBS chief Hester, and FSA chairman, Lord Turne, calling for action. "We need to know exactly what went wrong and what will be done to give us confidence that similar mistakes are not repeated in the future," says Tyrie. "Basic bank functions are crucial in a modern economy. It is deeply concerning that they appear to have been put at risk again."

Inexperienced RBS operative's blunder led to bank meltdown - The Register

Comments: (5)

A Finextra member
A Finextra member 27 June, 2012, 10:55Be the first to give this comment the thumbs up 0 likes

"The Royal Bank of Scotland (RBS) Group is an international banking and financial services company. From its headquarters in Edinburgh, the RBS Group operates in the United Kingdom, Europe, the Middle East, the Americas and Asia, serving over 30 million customers worldwide."

Having a single IT failure point for a global company of that standing (and, especially, if such a point is represented by "inexperienced (!) IT operative") is laughable (and doesn't sound convincingly plausible). If that is indeed the case, it's time to grind that axe.

A Finextra member
A Finextra member 27 June, 2012, 13:11Be the first to give this comment the thumbs up 0 likes

Situations such as this should never be attributed to the failure of an individual - its a failure of the 'System' that should have been designed to prevent a series of unfortunate incidents excalating into a major disaster...  Sadly it does appear that RBS are suffering from a condition that affects people who skip servicing or who cannot afford the best oil, filters and spares for their cars.... breakdowns are an inevitable consequence..... Someone at RBS may have taken a view that reducing costs, in order to securing a better bonus last year was a risk worth taking..... its a form of casino banking where you can win... but only if you dont lose (big time)

A Finextra member
A Finextra member 27 June, 2012, 16:11Be the first to give this comment the thumbs up 0 likes

Is this a case of:

Too many cowboys, not enough indians?

A Finextra member
A Finextra member 28 June, 2012, 09:14Be the first to give this comment the thumbs up 0 likes

What nonsense:

<< The FT quotes an insider at the meeting: "It was certainly an issue with this software. We will still have to establish if this was their fault or if it was our handling of the software." >>

Backups shoudl have been made, and it should have been able to revert back.  Also I've read that the upgrade was not properly tested.

The fault is most probably not with the outsourced ITer who managed to delete schedules whilst backing out - the fault lies with RBS management trying to offshore too agressively without ensuring knowledge transfer and bulletproof procedures to deal with such a situation.

They were offlien for a week, and as Keith Appleyard reports here on a Finextra blog, there are still severe consequences for customers.   Has anything like this happened before?  A top nationwaide bank going offline (more or less) due to an IT incident?

A Finextra member
A Finextra member 19 April, 2016, 16:49Be the first to give this comment the thumbs up 0 likes

As Chad would put it: Wot, no backup?

Trending