15 December 2017
Nigel Farmer

nigelrfarmer

Nigel Farmer - Software AG

19Posts 84,317Views 2Comments

Data, Data Everywhere, nor any Knowledge to Gain

27 October 2016  |  5081 views  |  0

The management, retention and reporting of huge data volumes is so onerous that many financial services firms are missing out on the value of this data; they are awash in a sea of data with nothing to drink.

Banks are dealing with ever increasing volumes of data, from trading data to communications and social media, coming from many different sources, in many different formats, residing in different silos. Regulations insist that they retain this data for longer, and to report more of it to the regulators.

And if individual banks are struggling, then one wonders how the regulators will cope with all the data they receive from all the banks and investment firms.

So how can you utilize this data, gain insight into it and act upon it? The value of data and the knowledge that can be gained from it diminishes over time, as does the opportunity for acting on it.

As an ecosystem for capturing, managing and deriving insights from extremely large collections of data, Hadoop stands out as a relatively low-cost choice. However, whilst tools for Hadoop can yield insights into patterns, trends and opportunities buried deep within large volumes of static historical data, an organization cannot use Hadoop operationally to take advantage of those insights in real-time. And one person’s view of real time may be considered ancient history on the trading floors, where time is measured and record in micro-seconds.

Streaming analytics platforms are capable of gaining insight on fast moving, large data sets from disparate sources, enabling you to act on it in real time. Leading platforms can cope with extremely high throughput with low latency, lending themselves to use in areas such as algorithmic trading, FX eCommerce, pre-trade risk, real-time fraud detection, market surveillance and anti-money laundering. Such technology could also be used to monitor transactions occurring on distributed ledgers and could have been used to detect or prevent recent high profile cases of hacking and fraud that have been seen with DAO and Bitfinex.

But there is still a place for Hadoop. In fact combing Hadoop with streaming analytics enables fast, actionable analysis of what is happening now, but also taking into account historical insight, such as trend analysis. Predictive models can also be defined based on historical data, and these models can be executed within the streaming analytics engine, providing a platform for historical, streaming and predictive analytics.

Such techniques are already being employed in areas such as FX, combining real-time analysis of what is happening in the market with historical trading patterns to provide customer tailored pricing. And market surveillance systems are also taking into account market activity combined with trader’s behavioral patterns and comparison with peers. As banks get a grip on their data, real-time analytics will play a key part in how banks operate and the services they provide to their clients. There is data everywhere, and there is also knowledge to gain from it. 

 

TagsRisk & regulationInnovation

Comments: (0)

Comment on this story (membership required)

Latest posts from Nigel

Are Smart Contracts getting smarter?

22 November 2016  |  4593 views  |  0 comments | recomends Recommends 2 TagsBlockchainInnovation

Data, Data Everywhere, nor any Knowledge to Gain

27 October 2016  |  5081 views  |  0 comments | recomends Recommends 0 TagsRisk & regulationInnovation

Capital Markets Firms Embrace Cloud Transformation

13 October 2016  |  10000 views  |  1 comments | recomends Recommends 0 TagsRisk & regulationInnovation

How to Manage Your IT Spaghetti

29 September 2016  |  3402 views  |  0 comments | recomends Recommends 0 TagsInnovation

CFTC Rules: Preventing the Rise of the Cybermen

14 September 2016  |  4145 views  |  0 comments | recomends Recommends 0 TagsSecurityRisk & regulation

Nigel's profile

job title Industry Director, Capital Markets
location London
member since 2016
Summary profile See full profile »
Nigel Farmer is the head of Capital Markets solutions at Software AG, responsible for the strategy,product management and marketing of solutions in areas such as Market Surveillance, Pre-Trade Risk, F...

Nigel's expertise

Member since 2016
19 posts2 comments
What Nigel reads
Nigel's blog archive
2016 (19)

Who's commenting on Nigel's posts