Blog article
See all stories »

Consolidated Audit Trail and really BIG Data

In New York the meeting was held today for the organisations that are interested in bidding to provide and run CAT NMS – the Consolidated Audit Trail system for the National Market System of the USA.

Keeping a full audit train of all securities operations across all market participants, being able to reconstruct what happened to every order, providing access to all market participants and to the many Self-Regulating Organisations (SROs) that monitor and regulate market activity, tracking instruments over years even when they change identifiers  – this is not just a “consolidated tape” project, but much, much more.

And the data volumes predicted are BIG.  The capability to handle 50 billion messages a day.  Storing all data for a 5 year period and enabling the market community and the regulating bodies to access all of that data.  All with a predicted growth rate in data volumes of 25% per year.

On top of the sheer scale there are the whole business operational issues of such a huge industry utility to address as well.  Customer Relationship Management capabilities to meet the needs of all regulated market participants and SROs.  Notification systems to advise market participants of issues, problems, faults, and all in a customer service-oriented timescale.

And then there’s the aspect of access: how will market participants reach this utility to interact with it not only in batch mode but in a real-time mode to deal with reporting, changes to reports, queries about transactions, etc.  And the question of standards – what messaging formats will be used to deliver data to and from and access data in this system that will be at the heart of the US securities market..

Today’s pre-RFP bidders conference saw the beginnings of what has to be one of the largest IT projects that has ever been undertaken in the financial markets sector anywhere in the world.


Comments: (1)

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 01 February, 2013, 14:17Be the first to give this comment the thumbs up 0 likes

Phew... 50B messages per day is H*U*G*E. To put that in perspective, UK FPS is rated for a peak volume of "only" 300M messages / day and Twitter handles "only" 175M messages a day. Besides, we can be sure that a good number of the 50B messages processed by CAT NMS will be about more than what someone had for lunch or dinner on that day, thereby multiplying the processing complexity! Given these figures, CAT NMS might prove to T*H*E largest IT project in any sector ever, period. That said, I noticed that the 50B figure on the CAT NMS website refers to the daily volume of records, not messages. Not sure if each message will contain only one record.