We were talking in the office recently about the pressure CEOs face when things go wrong. If a payment system falls over, and it hits the headlines, it can have a devastating impact on the business.

As a results, we thought it would be useful to list the top ten things that CEOs need to know about payment testing – and here they are:

Software testing should be a governance issue

High profile software failures have a significant impact on the share prices of those companies affected.  On average this equates to around 4% of the value of the business. If any other single element had this potential to damage your business value, you’d make it a governance issue.

Some of your people are ‘knocking together’ bits of code to test critical developments

Your organisation may spend hundreds of thousands, even millions on software solutions and then permit a contractor or software author to ‘knock together’ some code to test it.  To quote an old adage, you risk spoiling the ship for a ha’porth of tar. Project overruns in both time and cost put pressure on testing, and often force significantly shortened test cycles.  Without a properly engineered test environment you run the risk of your mistakes leading to full blown social media exposure.

The answer isn’t ‘throw more people at it’

The law of diminishing returns governs manual software testing in the same way it rules so many people-dependent functions.  F1 teams, faced with making cars faster, changed the shape, weight, balance and behaviour of their cars – they didn’t just add engines until they broke. Too often, adding people simply adds confusion, cost, inefficiency and management overheads, and rarely achieves the intended results.

You throw away a lot of the IP you develop

Manual testers come and go.  Either they leave, or you change your testing partner. They record their results in spreadsheets and written reports, which can’t be easily transferred, re-employed or automated. As testing evolves, the IP you develop is an important part of protecting your business. Not to capture this in an employable form is an oversight at best and can lead to added problems further down the line.

There is such a thing as human white space

Those queues you see at ATMs on a Friday lunchtime when traffic is busy are repeated in your test environment, while people wait for access to devices or schemes to complete their testing.  Limited equipment means literally hours wasted by testers waiting for availability – and that ignores the time taken to re-set the hardware and software environments when they do eventually become available.

Identifying a problem early saves a small fortune

Almost everyone in software development knows of the 1-10-100 rule. In essence, it costs a hundred times as much to resolve a software issue in production as it does in the design and development phases of a project.  In spite of this, testing remains an afterthought for too many organisations.

Your software is probably tested in pieces, and never in the way it works in the real world

It’s likely that if you take the simplest payment transaction in your business you have up to four test simulators, operated by people with specific skills, with separate results and different reporting formats. When, for instance, your ATM to VISA transaction fails its test, it will almost certainly require a handful of people to sit together in a room and work out where and why.  But in the real world a transaction flows. It doesn’t break itself into chunks that can be debated into an operational state. You should test as you run your business – from end to end; anything less than that is inviting failure.

Your test environment is almost certainly not auditable

Manual testing is inordinately difficult to repeat consistently.  Successful results obtained once may have depended on a unique set of manually created conditions that cannot be recreated.  It may be impossible to deliver proof that the test worked, or failed, when challenged weeks or months later, after a production outage for example.

Test to prove your software doesn’t work – not to prove it does!

Happy day testing is too tempting for developers.  If I’d been allowed to mark my own homework, I’d be a PhD. I’d have spent my time testing what I already knew, not what I had yet to learn.  Too many organisations focus on proving that software works, and yet the focus should be on trying to break it.  It’s too easy to be self-satisfied right up to the point that Twitter goes into meltdown because your clients can’t get at their money.

Your testing environment is probably not fit for purpose

Sadly, the test environments of most financial institutions are less than ideal. No matter how hard the staff work, their equipment is probably second rate at best and close to archaic at worst.  When I explained how testing works to a former senior product officer from a high street bank, he was shocked. He told me that if he’d known how his former company tested he would never have released a single product.

So what should you do?

You need to invest in the right testing solutions and the right test strategies. If your people don’t have the right strategy at their disposal, and the right environment within which to employ it, you are setting up both them and yourself for failure.

We’d suggest you get to know your test environment properly; learn what really happens and how exposed you and your organisation are. Testing shouldn’t be an afterthought or something you scrimp on – it should be central to ensuring that you reduce the risk of failure to an absolute minimum.