Understanding internal operations is crucial in financial services. Are public interfaces running smoothly? Are the back-end business systems as productive as they could be? Are infrastructure resources being allocated correctly based on business need? These are exactly the types of questions organisations should be able to answer but, surprisingly, struggle with.

However, a clear view into IT operations is often much easier said than done and it cannot be achieved with numerical or log reports alone. For real insight into an organisation's operations and systems,  we need to cut through the mountain of raw log data and turn it into visualisations and dashboards that make relationships clear to the human eye.

The Salamander

Spanish bank BBVA, needed a better understanding of its technical operations. The bank reached out to one of its innovation management divisions, BEEVA, to create a solution that would manage a wide variety of data and produce clear visual reports that could be quickly acted upon.

Called The Salamander, the tool has provided  the bank an unparalleled ability to optimise and simplify business IT processes, which ultimately saves costs and leads to an improved customer experience. Built on open source software the project has also given the bank significant savings by avoiding the expensive licensing costs of traditional software and increasing the pace of development.

BBVA executes thousands of batch jobs daily, structured in job chains with complex dependencies and a strict execution sequence order. To better understand it, BBVA required a workflow tool to visualise these dependencies so it could identify process enhancements and potentially reduce the number of jobs. The task was  big and complex and  required  integrating a number of next-generation technologies.

The Salamander team designed a solution running on a cloud computing architecture with MongoDB as the primary repository. MongoDB was chosen because it can scale large unstructured datasets across commodity servers in the cloud. Apache Pig and Hive execute data processing and transformation of raw data from the mainframe scheduler. MongoDB, which stores the processed data, works with a graph database to identify job relationships for faster network based queries such as minimum path between two given jobs. Finally, the application provides a graphical interface to browse the job catalogues. These generate data visualisations that clearly demonstrate the relationship between operations. The front and back-end of the application communicate via RESFUL APIs and NodeJS-based servers provide elasticity when accessing the stored data.

The tool is already helping the bank’s staff understand the complex network of batch jobs and processes that connect all of its services. The visualisations that Salamander creates is answering those crucial questions and helping to give BBVA an edge in the global banking industry.