Observability: The next phase in the evolution of data monitoring and breach investigations

The pivot by many banking and financial firms to digital by default over the past few years has played out against a huge surge in cybercrime, with the pandemic creating a perfect storm for new methods and types of attack. IBM reports that 23 per cent of all cyber-attacks are directed at financial institutions, while the total cost of a single data breach costs financial organisations $5.72 million USD on average. With the Log4j and Spring4Shell vulnerabilities now on the scene, security teams at financial organisations need to be able to successfully mitigate such threats while also preparing for what’s next.

At the same time, organisations undertaking these huge digital transformation projects often have highly dynamic, very complex, distributed environments. Tasked with managing and securing these initiatives, application developers, SREs and security teams often work in silos with competing incentives and no view over all applications. This is a huge challenge, given that the average enterprise has 10 or more tools for security and or analytics, and by 2025, will be managing 250% more data than in 2020.

So, how can financial services institutions get a better control over their data, and realise a more forward-thinking approach to security? The answer lies in observability.

What is observability?

Observability is a new topic and may not have appeared on some organisations’ radars yet. So what is it, and how is it different to traditional data monitoring? Put simply, observability is the practice of interrogating your environment without knowing in advance the questions you need to ask.

Gartner describes observability as “the evolution of monitoring your practice. It offers insight into digital applications, speeds innovation and enhances customer experience.” And it does that by letting people ask, ‘what if’ questions.

To build on this notion, observability pipelines drive operational efficiencies by getting the right data, to the right destinations – in the right formats – at the right time. As a result, businesses can more effectively realise digital transformation initiatives, slash costs and improve performance.

Observability is also more operational than analytics based. Instead of focusing on point-to-point solutions creating data silos, observability means taking all of the event data – logs, metrics and trace data – and running it through a strategic control point, putting the decision making about where this data goes back into the hands of the organisation. The user can decide how they want to route that data, to filter it, to redact data such as PII, meaning they can govern all of their data at one point. This way you can also reduce the amount of data that you are sending to downstream systems.

Additionally, with observability you can eliminate vendor or agent lock-in, simplify data engineering and regain complete control over your data. You can observe more – while paying less in license, storage and compute costs as you only forward the data you need to each tool.

Investigating breaches and threats

As well as contributing towards successful digital transformation initiatives, for financial services there is particular emphasis on security and the use of observability for the diagnosis of breach investigations and potential threats.

Observability provides the banking and financial services industry with an affordable way to retain more data for longer periods of time while still making that data easily accessible for breach investigations, whenever they happen.

Critically, observability pipelines allow financial institutions to place full-fidelity data in low-cost storage for as long as they need. As and when organisations discover a security breach, they can collect data from object storage and replay that security data to any SIEM or UEBA systems. This also means they can put customers at ease by quickly diagnosing and resolving existing breaches and potential cyber threats.

For privacy purposes, leading tools also enable organisations to encrypt or mask sensitive data in real time before it is forwarded to and stored at a destination, ensuring anonymity for every customer. This helps financial institutions keep customers’ personally identifiable information (PIN) safe – mitigating the risk of a breach and ensuring continued customer loyalty.

The pandemic forced many companies into digital transformation. There has therefore never been a time when the fates of IT and the business have been so tightly linked. In a digital first world, financial institutions need to ensure the reliability of their business-critical applications – even as they increase in complexity – and the security of the data these applications generate and is also entrusted to them by customers. Observability provides them with the huge advantage of understanding what’s happening in this enormously complex environment and, crucially, also enables them to provide a better, more secure, service to customers.

By Hash Basu-Choudhuri, General Manager, EMEA, Cribl