Regulators have budgeted for enormous spend on surveillance technology in the year ahead. On 31 March 2014, the UK’s Financial Conduct Authority revealed a 15% increase in its IT budget for 2014-2015, rising to £88.2 million from £76.4 million in 2013-2014.

On 4 March 2014 the Commodities and Futures Trading Commission (CFTC), the US derivatives market regulator, has proposed a 44% increase in its market surveillance budget for the fiscal year 2015, taking it up to US$53.4 million, from US$37 million in 2014.

The same day, the Securities and Exchange Commission (SEC), the US regulator for equity cash and derivative markets requested an increase of US$72.7 million to its budget for Compliance, Inspections and Examinations, up 26% to US$373 million dollars from US$300.6 million in 2014.

The regulators are keen to build/develop/buy systems that will enable them to pull data together from disparate sources and run some form of analysis on it to check for patterns of market abuse and fraudulent activity.

In some senses they are looking for a needle in a haystack. CFTC Commissioner Scott O’Malia was damning of the regulator’s capabilities when speaking at a SIFMA event on 19 March. He said “The problem is so bad that staff have indicated that they currently cannot find the London Whale in the current data files.”

Clearly investment is needed, but the question is, what sort of technology can best be used to support this sort of data analysis? Regulators in the US that I have spoken to are looking for a system that has the flexibility required to deal with multiple markets and products.

For example, the data that coming in from spot FX and swap markets will be enormously different, however market manipulation could involve trading on both. The SEC has already bought a system from an HFT firm that it uses to monitor publicly available market data, including cash equities, futures and options.

With budgets that big, there should be a lot of innovation on the cards.