Random thoughts about a recent computer glitch that hit TSX

Most financial systems are designed to handle outages in one or another way. However, even if redundancy is incorporated into the system’s architecture it is not uncommon to see that insufficient attention is paid to actually verifying that it will work out. In our practice, we have encountered situations when no more than eight hours (a single work shift) was allocated to check all possible failover combinations within rather complex systems.

Taking into account the level of competition in the industry, exchanges might put more effort into operational testing of their platforms and ensure that failover, monitoring and reconciliation tests are in place. 

Another point: in contrast to the crossing engines that ought to have a primary processing node and require architecture complexities to handle their failover, market data dissemination, by its very nature, can be implemented with relative ease in the active-active fashion. In this case it is not necessary to identify the exact chip responsible for the problem, it is sufficient to turn off the node. 

Anyway it is good to have additional insight for our exchange resilience whitepaper to be released shortly.