NoSQL to the Rescue

One of the most interesting, and potentially revolutionary, applications of big data technology in financial services has to be the schema-less trade repository. The best way to understand the solution is to first look at the problem.

In investment banks, there is a huge problem with data and system dispersion. By nature, the bank’s organization is highly siloed with the IT systems grouped by business lines and asset classes. An equities derivatives trader will trade on one system while a bond trader on another. Of course, since the data which has to be managed is particular to the asset traded, it makes sense that the trade capture systems and much of the front and middle office functions are specific for each asset class.

However, trades have to valued, P&L has to be calculated, risk has to be estimated, and accounting has to be completed for the whole bank in order to form a holistic view of the bank’s activity. So, the data from the different siloes have to be combined and forced into a universal model. Herein lies the problem.

To do this, investment banks have created ETL (extract-transform-load) processes throughout the IT architecture to move data from one system to another, modifying the data’s structure and granularity to fit the need of each downstream system. Over time, the banks have created a spider web of system-to-system links which is costly and inefficient to build and maintain.

This is where NoSQL can come to the rescue. Using NoSQL databases, data from different systems can be stored alongside one another, without the need to force them into a universal data model. This “schema-less” or “schema-on-read” architecture which underlies these databases removes the need to develop and maintain the multitude of ETL systems.

Moreover, a schema-less architecture also gives the platform flexibility to grow and evolve in time and thus enable quick innovation. In the old architecture, whenever there was a change in the upstream data format, the ETL layer, central data warehouse, and downstream systems had to be modified. Likewise, if the downstream system required new data (for new requirements), these data had to be added into the data stream with all systems requiring modification – alas, a very costly enterprise.

Working with products such as MarkLogic and HBase, we have implemented solutions for investment banking which have made first steps to revolutionize the business. By providing a flexibility and agility to improve the bank operations, rationalize the bank’s architecture, and reduce on-going IT costs, NoSQL allows us to tackle IT problems in a new, better way.