Time series data is unique as it accumulates more quickly than other types of data because of its nature: each record is a new record, not an update or replacement. With this influx of time series data at a rapid rate, storing and querying data can become problematic. Relational and NoSQL databases are not optimised for such extremely large datasets with the same extent of analytics capabilities; time series databases (TSDBs) are needed as they can handle higher ingest rates, faster queries at scale and can better prepare time series data for analytics by bucketing and visualising data more efficiently.

To unlock the value of time series data, organisations must be able to store data that accumulates quickly and query it in a performant way. Capital markets firms utilise vast amounts of historical and streaming data to perform real-time analytics and inform decision-making. Whether they are predicting stock prices and exchange rates or projecting the net asset value of funds and net capital requirements, these institutions are migrating their time series databases to the cloud for greater flexibility, scale, cost optimisation and agility.

Finextra Research spoke to Balaji Gopalan, senior solutions architect, financial services partners; John Kain, worldwide business and market development for banking and capital markets and Vera MacLeod, project manager, worldwide business and market development for capital markets at Amazon Web Services about why time series data is critical to capital markets firms and how the cloud can amplify the benefits of time series databases.

Why is time series data critical to capital markets firms?

From the trading desk to the back office, time series data analyses drive many critical use cases for capital markets firms. A time series database using legacy infrastructure cannot keep pace with their primary needs today: adapting immediately to market conditions and getting investment models quickly to market.

According to Kain: “Every investment strategy has a limited lifetime. When it comes to trading and trade analytics, the advantage goes to firms that can most quickly leverage and analyse time series data for price and demand forecasting, algorithmic trading, back testing and transaction cost analysis.”

Gopalan specifies: “When analysing cross-asset prices, along with macro and market events, global news and social media sentiments, capital market firms are not only looking for correlations, but also for causality.” He adds: “Time strings these seemingly disjoint sets together to drive sharp insights. Even when analysing why a trading or algo engine made a specific decision at a point in time requires stitching data across the time line.”

Time series data also can be used for risk assessments - whether it is credit, market or counterparty - stress testing and back testing risk models. It is also utilised for transaction surveillance where order and execution data is visualised to monitor trader behaviour, identify anomalies and detect potential market manipulations.

What are the requirements to manage time series data?

Capital markets firms have always needed to conduct calculations for risk management, regulatory compliance, product development and pricing, as well as clearing and surveillance, but the volume of these calculations has increased at a dramatic rate. As a result, firms are migrating their on-premises compute farms to the cloud where they can leverage both the on-demand capacity and scale cloud affords.

They are also using cloud-based data lakes to store information and build a foundation for analytics. These data lake architectures allow firms to better ingest large data volumes and deploy the right tools for their specific analytic needs. Time series databases in the cloud help surface insights and make decisions even faster, however, time series analyses are resource-intensive workloads.

“Time series data requires fast in-memory array processing, easy query capabilities, scalable user access and large storage capacity,” Gopalan states. “Highly secure, scalable, reliable and performant infrastructure is necessary to run time series analyses.”

Without these capabilities, the sheer volume of time series data creates an immediate ingestion challenge for capital markets firms. Building new trading scenarios, for instance, requires capturing market data. The data necessary to just maintaining a full order book can exceed several terabytes per day.

“The fact of the matter is that most other databases are less performant when it comes to time series data,” MacLeod states. “Relational databases do not scale for the volume of time series data, while NoSQL databases do not organize time series data efficiently for time-focused analytics.”

To cope with the huge volume of data when using these types of databases, administrators often divide the data into different databases and/or load data directly into their analytics. This approach, Macleod highlights, results in artificial data silos and increases the complexity of and strain on an organization’s infrastructure.

On the analytics front, these databases have minimal capabilities. Even simple time series operations such as “as of” joins or resampling are challenging to implement with relational and NoSQL databases.

“Time series databases deliver the fastest ingestion rates with efficient storage by compressing data, preserving its coherences and storing it in a form that dramatically simplifies further analysis,” Gopalan explains. “They remove the need to manually split the data and allow transparent querying of the complete dataset. High querying speed ensures that as these databases ingest gigabytes of data, they can answer queries at very low latency.”

What are the benefits of a time series database?

• Handle scale by introducing efficiencies that incorporate time as a key component
• Efficiently store and present many time points for each element
• Have higher ingestion rates and faster high-performance queries
• Include functions that enable continuous queries and flexible time aggregations that improve user experience
• Offer custom time series functions optimised for performance
• Are built to view data in a single platform

How can the cloud amplify the benefits of time series databases?

Migrating time series databases to the cloud provides access to virtually unlimited infrastructure, including on-demand burst capacity. This enables financial institutions to use thousands of cores for short periods of time, with the potential of achieving low network latency and high network throughout.

MacLeod comments: “In order to conduct meaningful analyses, trading models are testing years of data. When there is limited capacity, analysts often compete and some are not able to act on new signals or test their research ideas.”

Gopalan specifies: “Time series databases in the cloud are ideal for specific and intermittent compute-intensive workloads such as on-demand risk calculations in reaction to market events in real time. Financial institutions can offload their data center overhead, focus resources on their core differentiators and boost performance of their workloads.”

Managed services from cloud providers such as AWS and their partners make it easy to load, store and analyse time series datasets as they offer storage that can handle transaction-intensive workloads, tools for real-time analysis and data streaming capabilities to capture events as they occur.

Capital markets firms are adopting the AWS cloud in order to discover new opportunities, rethink and redesign operating models, and implement cost-saving measures that increase efficiency.

Download this eBook to learn through use cases how capital markets customers are working with AWS to accelerate their speed to market, strengthen security, enrich customer experiences, and make better data-driven decisions.