14 December 2017


Retired Member

3,251Posts 11,853,906Views 3,524Comments

Consolidated Audit Trail and really BIG Data

29 January 2013  |  4115 views  |  1

In New York the meeting was held today for the organisations that are interested in bidding to provide and run CAT NMS – the Consolidated Audit Trail system for the National Market System of the USA.

Keeping a full audit train of all securities operations across all market participants, being able to reconstruct what happened to every order, providing access to all market participants and to the many Self-Regulating Organisations (SROs) that monitor and regulate market activity, tracking instruments over years even when they change identifiers  – this is not just a “consolidated tape” project, but much, much more.

And the data volumes predicted are BIG.  The capability to handle 50 billion messages a day.  Storing all data for a 5 year period and enabling the market community and the regulating bodies to access all of that data.  All with a predicted growth rate in data volumes of 25% per year.

On top of the sheer scale there are the whole business operational issues of such a huge industry utility to address as well.  Customer Relationship Management capabilities to meet the needs of all regulated market participants and SROs.  Notification systems to advise market participants of issues, problems, faults, and all in a customer service-oriented timescale.

And then there’s the aspect of access: how will market participants reach this utility to interact with it not only in batch mode but in a real-time mode to deal with reporting, changes to reports, queries about transactions, etc.  And the question of standards – what messaging formats will be used to deliver data to and from and access data in this system that will be at the heart of the US securities market..

Today’s pre-RFP bidders conference saw the beginnings of what has to be one of the largest IT projects that has ever been undertaken in the financial markets sector anywhere in the world.

TagsTrade executionPost-trade & ops

Comments: (1)

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 01 February, 2013, 14:17

Phew... 50B messages per day is H*U*G*E. To put that in perspective, UK FPS is rated for a peak volume of "only" 300M messages / day and Twitter handles "only" 175M messages a day. Besides, we can be sure that a good number of the 50B messages processed by CAT NMS will be about more than what someone had for lunch or dinner on that day, thereby multiplying the processing complexity! Given these figures, CAT NMS might prove to T*H*E largest IT project in any sector ever, period. That said, I noticed that the 50B figure on the CAT NMS website refers to the daily volume of records, not messages. Not sure if each message will contain only one record.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Comment on this story (membership required)

Retired's profile

job title
member since 2014
Summary profile See full profile »

Retired's expertise

Member since 2009
3181 posts3,524 comments
What Retired reads

Who's commenting on Retired's posts

Ketharaman Swaminathan
Raymond Lee
James Andrew
Dharmesh Mistry
David Andrzejek
Ralf Ohlhausen
Tom Hay
Nicola Cowburn
Michael Wright
Charmaine Oak
Francis Chlarie