THE BIGGEST SCANDAL IN SOFTWARE DEVELOPMENT

Many years ago I was invited by a UK FTSE 100 company to a meeting to discuss a project to audit one of their legacy applications and to make recommendations about how they could redevelop it. This application had been operational for over 10 years and was coming to the end of its life.

One of the things that I discovered was that one of the original programmers had put in a logging mechanism that counted the number of times each part of the system was used.

Boy, what an eye-opener that was!  On examination I discovered that just over 10% of the modules in the system had never been used since the system went live, over 10 years previously. A further 14% of the modules had only ever been used once. Beyond that there was a whole raft of modules (nearly 25% of modules) that had been used less than a dozen times. In total, close to half of the system had little or no usage.

At the time I thought that this was an isolated case but in subsequent years I discovered that, far from being isolated, this was a norm in the software world. In fact this is now a widely known phenomenon that is confirmed by further research.

In 2001 the IEEE in their paper “Improving software investments through requirements validation” found that in 400 projects examined over a 15 year period, less than 5% of the code was actually useful or used.

A 2002 Standish Group study of software projects found that 45% of features were never used and only 20% of features were used often or always. Another study by Dupont claimed that “only 25% of a system's features were really needed”.

Whilst there are plenty of people ready to argue the accuracy of these numbers there is nobody arguing that the underlying principle is true – that we build vast amounts of software that never gets used.

In addition to my own personal experiences I’ve also had many conversations over the years with CIOs and other technical people and they have all had their own tales to tell of wasted development effort. It is happening on every project in every organisation around the world.

If you let your mind wander for a few seconds and you extrapolate all this data out then you could be forgiven for arriving at an incredible conclusion - that around half of ALL software that is running today is not used. Given the amount of software that now exists in the world this represents trillions of pounds/dollars/euros spent on software that is not needed. What a waste!

This is mind-boggling stuff. But it gets even more mind-boggling. We are still creating waste at a phenomenal rate. And as waste increases the costs increase on a non-linear basis due the increased complexity of the system.

In 2008 Grady Booch, an IBM chief scientist, said that "Every advance for the future state of the world requires the presence of software yet to be written". We will therefore be writing more and more software way into the future.

So in the coming years further trillions (quadrillions?) will be spent on new software and further trillions of waste will be created unless we stop paying to have software developed that we don’t need.

If the above doesn’t motivate you to do something about the waste in your organisation then ponder this. You could cut your software development budget in half if you only built the features that were really needed. You would get the software in half the time and get the benefit of the software earlier. That’s all going to feed into the bottom line.

A LESSON IN CONTROLLING WASTE

You might think that as a custom software developer we would relish the thought of creating more and more software. On the one hand I am delighted if our services are more in demand because of the continuing need for more software. On the other hand however, I abhor the thought of adding to the software waste heap. It’s just not necessary. In fact it makes our job even harder as more code means more complexity.

One of the most successful projects I’ve ever worked on was for the insurance subsidiary of a major UK high street bank. The reason I mention this is because I can quite confidently say that we created little or no waste. There just wasn’t time to do it.

We got the call from the client 8 weeks before a new legal deadline was due to come into effect. They needed a new system to be in place by the deadline. Could we do it?

We said that we could definitely build a system in that time but there was no room for delays and no room for extraneous features. The system would have to be pared to the bones. They understood and so we were given 8 weeks to design, build, test and implement the system.

We elected to use an Agile Development approach to the project as it was the only way that it would get done on time. At the time our client was not familiar with this approach but they quickly embraced it when we started working with them.

We successfully delivered the system on time to great thanks and relief from our client. What was really interesting about this project were the behaviours that were exhibited by all sides that were essential to its success.

We had a shared goal

People on both sides, our team and theirs, fully understood that the deadline was immovable and that compromises would need to be made in order for us to hit target. Our developers were closely involved in deciding what could be built in the time available.

Far too often I’ve seen projects start off with an agreed deadline which then starts to move as it emerges that it is more important to have more features (and waste) over an on-time delivery.

There was no room for egos or politics

There were no politics or egos in play. Across departments everybody worked towards the same goal. It was a motivating and collaborative experience for both sides.

There was a high degree of trust

Our client quickly saw that we were just as committed to the deadline as they were. We were prepared to put in extra effort to squeeze in features that they needed. On the other hand they also trusted us when we said a feature needed to be postponed as it wouldn’t be delivered on time.

Decisions were made fast

Decisions were made fast. Requested features were dropped if the group decided they were not “must-haves” or if they threatened the deadline.

One “must-have” at the start of the project was the need to have integration with postcode data. This got dropped because the lead time then for acquiring the file was about 4-6 weeks. We wouldn’t have been able to build the interface, test it and integrate it on time.

Success

The system successfully went live without postcode integration and it was added in a second release a few weeks later. The system went through a number of further releases with the feature set only growing in response to how the system was actually being used. There was little or no opportunity for waste to creep in.

HOW DOES WASTE GET INTO SOFTWARE?

There are many moving parts and processes involved in a software development project. Waste can occur at all points but here are 4 areas where I’ve regularly seen waste creep in.

Users with no buy-in to the budget and/or timescale

If the users face no consequence then why would they be motivated to keep the requirements under control? As far as they are concerned they want every feature that they can think of.

Without a strong product champion, who is committed to, and accountable for, the budget and/or timescale more than the feature set, you will certainly create waste.

Having only a single release

This is one I see time and time again and it just creates the wrong behaviour patterns. If users know that they are only going to get one shot at it they will pile in the requirements and it will be next to impossible to control waste.

Taking a big bang approach to development is such a flawed way to work but it is prevalent throughout the corporate world. If I could have one wish for this article it would be that I hope that it alerts some senior people to the vast amounts of time and money wasted on software development and that they review their approach to their projects.

A dominant character deciding what’s needed

I would rate this as one of the most pernicious ways that waste can get into a system. These people are very insistent about what features are “must-haves” without regard for the cost or time involved. Often they will insist that they know what other users want without even checking if that is the case.

Unlike a good product champion, there is usually little consensus in their decisions and they will usually ignore the fact that they are demanding more than can be achieved for the budget or in the time available.

These people invariably create lots of waste. And then what happens when they either move departments or leave the company? Does anybody else use those features? Does anybody even know those features exist?

Developers inventing requirements and/or complexity

What? I hear you say. Surely not? How could that happen? Well yes it does happen and far more often than you might imagine. Younger, less experienced developers are particularly prone to this but it can happen across the development spectrum.

Developers are prone to having a “just-in-case” mentality. They will add extra features or capabilities just in case the users need it or because they imagine that the user must want it.

They are also prone to writing code that is far more complex than is necessary. Object Oriented Languages (which we use) have a lot to answer for here.

Some developers go on a mission to abstract their code to the nth degree because that’s what the purists say should be done. However, purists tend to live in Nirvana where there are no deadlines, there’s infinite money for development and they don’t have to maintain and support their complex code. If only they would listen to Albert Einstein – ““Make things as simple as possible, but not simpler.”

HOW DO YOU AVOID WASTE IN THE FUTURE?

You could probably fill a library with all the books available advising on all sorts of development practices designed to make the process better, leaner, quicker, cheaper. However, for the purposes of this short article I would offer the following four suggestions.

Adopt an incremental approach to development – No more big bangs

Commit to delivering only the most important 25%-30% of features in the first release. Get that released and then listen to the feedback. Make the second release contingent on the feedback from the first.

By responding to what’s actually needed, this will greatly increase your chances of not creating waste. It will also create a system that the users actually want to use.

If you think about it, good software will usually change people’s working practices and behaviours. They will be able to do things they couldn’t do before and they will become more efficient at certain tasks. It’s only through the use of the system that you can really know what the most valuable features for the next release are.

Make sure the users know that there are further releases to come and that you will base the decision on what to release on their feedback. This gives you a much greater chance of controlling the “kitchen-sink” mentality.

Use timeboxing to control scope (& waste) creep

This is the highest payback action you can take to avoid waste. Scope creep is the cancer in software development that leads to huge amounts of waste.

Think of the project I mentioned above that had an 8 week deadline that was immovable. The feature set was controlled by the deadline. There was no room for waste to creep in.

If you’ve never worked this way before it can be hard to believe that you will get the system you really want. And that’s true because instead you will get the system you really need and without any waste!

Use an Agile development method

If you can get the first two suggestions implemented then you will need to have a development approach that is compatible with these. This is where Agile development fits perfectly.

Agile development is based around the concept of frequent deliverables where the requirements can change as the project progresses. It is also founded on the principle that adherence to timeboxes matters more than scope.

Don’t tolerate obscure or complex code

To deal with this issue you will need both an educational and a monitoring approach.

You should have rigorously enforced coding standards which ensures that developers follow the corporate line. You should also implement peer reviews of the code. It is vital that you have extra eyes on the code to ensure that unnecessary complexity doesn’t creep in.

To ensure that your developers are consistently improving you should develop a best practice culture in your organisation. Make it unacceptable to develop obscure code. Get your best experienced developers to mentor more junior people and set the quality expectations.

SUMMARY

Every organisation, large and small, has already wasted, and continues to waste, huge amounts of time and money on unnecessary software. The key question is this. Now that you know how much waste there is in software what do you plan to do to stop it?