19 April 2014

Chris Pickles

Chris Pickles - Consultant

87 | posts 189,850 | views 15 | comments

Consolidated Audit Trail and really BIG Data

29 January 2013  |  3337 views  |  1

In New York the meeting was held today for the organisations that are interested in bidding to provide and run CAT NMS – the Consolidated Audit Trail system for the National Market System of the USA.

Keeping a full audit train of all securities operations across all market participants, being able to reconstruct what happened to every order, providing access to all market participants and to the many Self-Regulating Organisations (SROs) that monitor and regulate market activity, tracking instruments over years even when they change identifiers  – this is not just a “consolidated tape” project, but much, much more.

And the data volumes predicted are BIG.  The capability to handle 50 billion messages a day.  Storing all data for a 5 year period and enabling the market community and the regulating bodies to access all of that data.  All with a predicted growth rate in data volumes of 25% per year.

On top of the sheer scale there are the whole business operational issues of such a huge industry utility to address as well.  Customer Relationship Management capabilities to meet the needs of all regulated market participants and SROs.  Notification systems to advise market participants of issues, problems, faults, and all in a customer service-oriented timescale.

And then there’s the aspect of access: how will market participants reach this utility to interact with it not only in batch mode but in a real-time mode to deal with reporting, changes to reports, queries about transactions, etc.  And the question of standards – what messaging formats will be used to deliver data to and from and access data in this system that will be at the heart of the US securities market..

Today’s pre-RFP bidders conference saw the beginnings of what has to be one of the largest IT projects that has ever been undertaken in the financial markets sector anywhere in the world.

TagsTrade executionPost-trade & ops

Comments: (1)

Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 01 February, 2013, 14:17

Phew... 50B messages per day is H*U*G*E. To put that in perspective, UK FPS is rated for a peak volume of "only" 300M messages / day and Twitter handles "only" 175M messages a day. Besides, we can be sure that a good number of the 50B messages processed by CAT NMS will be about more than what someone had for lunch or dinner on that day, thereby multiplying the processing complexity! Given these figures, CAT NMS might prove to T*H*E largest IT project in any sector ever, period. That said, I noticed that the 50B figure on the CAT NMS website refers to the daily volume of records, not messages. Not sure if each message will contain only one record.

Comment on this story (membership required)
Log in to receive notifications when someone posts a comment

Latest posts from Chris

China joins the global LEI initiative

12 August 2013  |  2055 views  |  0  |  Recommends 0 TagsSecurityRisk & regulation

Venture Capital, Risk Management and the Dealing Room

13 June 2013  |  2257 views  |  0  |  Recommends 1 TagsSecurityRisk & regulation

Chicago - where the Big Data grows

25 March 2013  |  3497 views  |  3  |  Recommends 0 TagsTrade executionPost-trade & ops

25 years of the German Exchange

22 March 2013  |  2165 views  |  1  |  Recommends 0 TagsTrade executionPost-trade & ops

Vote for FIX for LEIs and CAT NMS

08 February 2013  |  2500 views  |  1  |  Recommends 0 TagsTrade executionPost-trade & ops
name

Chris Pickles

job title

Consultant

company name

Consultant

member since

2009

location

England

Summary profile See full profile »
I help organisations that work in the financial sector around the world to understand better how ...

Chris's expertise

What Chris reads
Chris writes about

Who is commenting on Chris's posts