The management, retention and reporting of huge data volumes is so onerous that many financial services firms are missing out on the value of this data; they are awash in a sea of data with nothing to drink.

Banks are dealing with ever increasing volumes of data, from trading data to communications and social media, coming from many different sources, in many different formats, residing in different silos. Regulations insist that they retain this data for longer, and to report more of it to the regulators.

And if individual banks are struggling, then one wonders how the regulators will cope with all the data they receive from all the banks and investment firms.

So how can you utilize this data, gain insight into it and act upon it? The value of data and the knowledge that can be gained from it diminishes over time, as does the opportunity for acting on it.

As an ecosystem for capturing, managing and deriving insights from extremely large collections of data, Hadoop stands out as a relatively low-cost choice. However, whilst tools for Hadoop can yield insights into patterns, trends and opportunities buried deep within large volumes of static historical data, an organization cannot use Hadoop operationally to take advantage of those insights in real-time. And one person’s view of real time may be considered ancient history on the trading floors, where time is measured and record in micro-seconds.

Streaming analytics platforms are capable of gaining insight on fast moving, large data sets from disparate sources, enabling you to act on it in real time. Leading platforms can cope with extremely high throughput with low latency, lending themselves to use in areas such as algorithmic trading, FX eCommerce, pre-trade risk, real-time fraud detection, market surveillance and anti-money laundering. Such technology could also be used to monitor transactions occurring on distributed ledgers and could have been used to detect or prevent recent high profile cases of hacking and fraud that have been seen with DAO and Bitfinex.

But there is still a place for Hadoop. In fact combing Hadoop with streaming analytics enables fast, actionable analysis of what is happening now, but also taking into account historical insight, such as trend analysis. Predictive models can also be defined based on historical data, and these models can be executed within the streaming analytics engine, providing a platform for historical, streaming and predictive analytics.

Such techniques are already being employed in areas such as FX, combining real-time analysis of what is happening in the market with historical trading patterns to provide customer tailored pricing. And market surveillance systems are also taking into account market activity combined with trader’s behavioral patterns and comparison with peers. As banks get a grip on their data, real-time analytics will play a key part in how banks operate and the services they provide to their clients. There is data everywhere, and there is also knowledge to gain from it.