Testing.  Everybody responsible for trading systems knows they need to do more of it.  Few have the time to do it right.  Fewer have the resources to commit to it.  The electronic trading industry has suffered a rash of system failures over the past few years that have each pointed to the need for more testing.  Time and again it has been shown that the risk of not investing in testing is too large to ignore, yet this process still goes overlooked and has gone largely unchanged for close to 20 years.

Industry experts agree that automating the testing process is the most cost-effective approach, although it usually requires a large upfront investment. Few organizations have the internal resources to implement it properly.

 

The issues with automated testing

Those who have built substantial automated test environments know that automated testing is not always a silver bullet.  The hardest part is finding a team with the expertise to develop a suite of automated tests.  These individuals must have great knowledge of the domain, a grasp of programming concepts, and experience with scripting.  Testers with a combination of these skills do not stay in testing roles for long.  Outsourcing or bringing in consultants could be a good option, but the specialized domain knowledge and familiarity with your environment walks out the door as soon as the project is over.

An often overlooked area of automated testing is the cost associated with maintenance.  The maintenance costs of hosting and developing an automated test environment can creep in from many places.  Trading systems are always changing, driven by new regulations, functionality, or customer requirements.  Unless a good process is in place to identify new use cases that should have automated tests, the test suite may become stale quickly.  Test data is another source of maintenance costs.  Maintaining static data like instruments and accounts is straightforward for equities.  For those working in futures, options, or fixed income you have to deal with expiring contracts and other date-related events.

For firms routing orders, market data is another point of contention.   There are cases where prices are driven by live market prices, or data from external test environments.  In the best case this market data can be controlled within your own test environment.  More often pricing data is subject to counterparty systems such as exchange test environments.

At the end of the day, a simple suite of functional tests is great for catching simple bugs in calculations and field mapping.  The defects that are difficult to reproduce, such as race conditions and concurrency issues, need a completely different approach to automated testing.  “Flash crashes do not result from glitches, evil spirits, or sunspots.” says Bob Binder, author of Testing Object-Oriented Systems. “They happen when an unusual pattern of trading volume and message flow provokes a latent behavior that provokes even worse latent behavior. Traditional testing approaches, even stress testing, cannot trigger this kind of failure. But, high-volume and high-variation test suites, running in a realistic test environment, can.”

 

The bright side

When it comes to automated testing in the electronic trading industry, we are quite fortunate.  The business cases are well-defined.  There are standard APIs like FIX that are easy to drive test cases through.  The question you have to ask yourself is why is automated testing not easier than it currently is?

The answer is that it not only could be easier - it should be easier.  There are many areas where efficiencies can be gained.  Most test automation projects in our industry entail rewriting the same tests over and over again.  The differences between similar organizations cover less than 20% of the tests that are written.  Should you really pay a consulting firm to rewrite the same test they have been writing at different organizations for the past decade?

 

Continuous Testing

Electronic trading firms have achieved productivity gains by employing continuous improvement processes like Agile and Continuous Delivery.  In a similar vein, continuous testing is needed to ensure the quality of our trading systems.

A continuous testing strategy for electronic trading takes into account changes in trading system behavior, test data, and counterparty behavior.  In this way the tests are always up to date and comprehensive.  Operators constantly receive feedback as to the quality of their trading system.

Continuous testing in the electronic trading industry is achievable through the combination of well-defined business cases and easy to test interfaces.  Having well-defined business cases provides the ability to automate the full testing lifecycle.  Identifying system behavior, writing tests and managing test data can all be automated.  Having standard messaging APIs like FIX makes simulating counterparty behavior and virtualizing services possible.

Those responsible for reliable operation of trading systems should expect more value from their testing services and use higher levels of automation to improve system reliability and integrity and reduce operational risks.  The highest value, lowest risk and most sustainable approach is continuous testing.  Continuous testing automatically takes into account changes in trading system environments over time and provides a greater breadth and depth of testing than any other approach to testing.