Forget big, think data

The industry is buzzing with talk about ‘big data’, but what underpins the hype and what does big data bring to the world of application development within investment banking?

Since the inception of big data, designing applications has become a lot more complex. In the past if you needed a database, Oracle would usually be your first choice and, if not, the standard choice would certainly have been to employ a Relational Database Management System (RDMS) using Structured Query Language (SQL). Pre-big data, a ‘one size fits all’ approach to application development was the industry norm.

The extreme simplicity of this standardised approach has provided many benefits to financial services organisations; developers know how to use it; database administrators know how to support it; specialists have learned to work within its parameters to create highly tuned environments; a set of analytical tools that assume the relational database model has been made readily available by tool developers; and enterprise licenses have been acquired by purchasing departments for these widely employed solutions.

Despite the benefits it seems that for certain applications, such as those that have either massive data sets, many complex relationships, or relatively unstructured data, the standard ‘one-size-fits all’ approach did not work well enough. The huge data volumes and vast user population handled by companies such as Google and Amazon have forced them to rethink their data management approach. As a result they have created alternative database models that support massive data volumes using techniques that provide high performance and availability over clusters of many machines, in multiple locations.

Whilst there are many benefits to using existing relational databases, developers have struggled to align the paradigms used in modern application design with persistence in a relational database. Architects have been constrained by the limitations of relational databases, unable to explore a rich set of database technologies that could be used to create well designed and innovative applications.

The big data revolution has fuelled the emergence of a variety of database models, often grouped under the banner of ‘NoSQL’ which in its broadest sense, can be interpreted as being anything other than a relational database. Although you may not have a big data ‘problem’, by no means should you sit back and watch as others implement big data technologies. Rather than focussing on whether or not we have a big-data problem and how to solve it, we should be excited about the broad range of application development opportunities available to us now, thanks to the emergence of big data tools.

However, adopting new technologies is often difficult and never without drawbacks, especially in the controlled and regulated world of investment banking. Challenging Oracle’s long established dominance (based on the simple customer confidence gained from the fact that it underpins thousands of production applications) is a task that should not be taken lightly. However, the difficulties involved in doing this should not prevent concerted exploration into alternative database models and their effect on application design.

This mammoth task cannot be completed by technologists alone since their primary role is to understand emerging technologies and the benefits they may bring. As has always been the case, technologists then need to work with the wider business to understand the application requirements and to produce the best design, having considered all functional and non-functional business / user needs. With the increase in data management tools available, greater interaction between technologists and the business is essential in order to produce optimal solutions.

So, let’s not focus on big data as the only current technology trend and go relentlessly looking for problems to solve in order to use this exciting new toolset. Instead let’s understand the additional database models that now exist so that we can design applications that most effectively support the data requirements of our applications.

To put it simply, let’s forget big and think data!