It’s crunch time for the Big Data crunchers.

Having lavished millions of dollars on data scientists to search for patterns in a deluge of digital information, banks and financial institutions need to start doing something to make that newfound intel pay. The question is, what?

Big Data - the bits and bytes of everyday life harvested from our increasingly digital world - has been hailed as the foundation of a new financial architecture. Predictive analytics, machine learning and artificial intelligence (AI) are the mechanisms most commonly touted to realise that vision.

But aside from sales traders using predictive analytics to help clients with their decision processes, some well-publicised robo-advisory services, and a handful of AI-managed funds, institutions have yet to deploy on any scale the means of capitalising on their newly bulging databases.

Having new data-driven insights is great, but operationalising and monetising them is the tricky part. Fundamentally, it begs the question of how firms in the financial markets sector can most effectively use all of the data they have gathered to make smarter decisions.

There are gnawing indicators that Big Data’s promise is being regarded in some quarters as little more than hype. Chiron Investment Management for example, is among hedge funds that have begun to wonder whether data-led investment strategies - such as smart beta - really offer better value than traditional ploys. Other asset managers are asking if too much confidence has been placed in Big Data’s ability to generate new markets.

Turning Big Data into business value

On the other hand, investment firms such as Vanguard and Blackrock are investing heavily in data-driven robo-advisory platforms. Tokyo Stock Exchange is one of an increasing number of market operators using AI techniques to detect unfair trading.

Regardless of whether the use of Big Data and AI in financial markets is overhyped or not, few doubt that data scientists will discover some amazing things in the gush of information they’re receiving. This in turn will unlock a wealth of new opportunities.

Firms can get a better insight into what makes their customers tick and personalise products accordingly.

• Predictive analytics can assess the market impact of trade executions, helping firms preserve alpha by enhancing execution strategies.
• Machine learning can be used to improve and streamline operations by resolving exceptions before they happen, enabling more intelligent STP (straight-through processing).
• Aggregated data can give clues to new sources of revenue and suggest pointers to asset managers looking for new investment strategies.
• Natural Language Processing can automate workflow from written rules and regulations, helping companies navigate increasingly choppy regulatory waters.

Speed to Market

However, to make Big Data work in these examples and others, firms need the right kind of platforms in place to enable them to put these ideas into production quickly, by integrating them into their operational IT systems.

Just as importantly, those processes need to be future-proofed. Once opened the data gusher will not stop and it will continue throwing up new and wonderful insights. For that reason, the platforms that companies put in place have to be flexible and adaptable, able to respond quickly to changes in the patterns and complexity of data.

Traditional quantitative and systematic hedge funds such as Two Sigma and Winton are already starting to evolve experimental funds built on “deep learning” systems that try to replicate the neural networks of the brain. In both examples, time is needed for the networks to digest enough data to be able to make calculated investment decisions.

For too long Big Data has looked like a great answer in need of a tough question. But only with the right platforms put in place — and soon — can it show us its true potential.