19 December 2014

Software Development Expert

Paul Smyth - Kynetix Technology Group

5Posts 44,881Views 2Comments

Business Knowledge for IT

This community aims to provide links, resources, book suggestions, tips and insights to facilitate learning and development of IT professionals in financial services, and to develop a forum for IT professionals to exchange views on various related items.

7 Reasons Why Software Development Is So Hard

08 August 2012  |  24926 views  |  6

Introduction

I’m often asked by lay people why we humans can undertake large construction or engineering projects with relative success and yet we struggle to deliver software projects without bugs and on time.

In an attempt to explain why this is the case I give below 7 reasons why software development is very difficult. This is not an attempt to condone shoddy software development practices. Rather, I’m trying to show that even with excellent development processes it is still difficult to do software development right.

1. The software industry is young

Humans have been building house, roads and bridges for thousands of years. We’ve no idea how many house or bridges collapsed in the early days as humans learned the correct techniques for building these structures.

One has only to look at the infamous Tahoma Narrows bridge collapse in 1940 to realise that, thousands of years after the first bridges were built, they still hadn’t perfected bridge building.

In comparison the software industry is only about 50 years old. We still have a long way to go before we have the body of experience behind us that the construction and manufacturing industries have.

Today the construction industry use mostly prefabricated materials and parts. Most of these are made by machine and have been tried and tested on many other projects.

The software industry on the other hand doesn’t have the range of pre-built components that other industries have. Software systems are fundamentally built by a process of discovery, invention, and creation of new components with the result that each new system is a custom project created from scratch. This leads us to our next point.

2. Every line of code is a potential point of failure

Because all new projects are custom built it follows that every line of code is unproven and therefore should be tested. However, in the real world, this is totally impractical.

Each line of code will have dozens, even thousands, of possible inputs, outputs, states or dependencies to deal with. It can impact, or be impacted by, other lines of code or by external factors. Even if it was possible to document every test case for a line of code you still couldn’t be sure that there wasn’t some unknown factor that could cause an error.

And testing a single line of code is only part of the challenge. No line of code exists on its own. It is part of the whole system and the whole needs to be tested to ensure that all parts of the application function correctly.

The sheer complexity of software means it is impossible to test every path so in the real world the best project teams will implement processes that are designed to increase the likelihood of the software being defect free. They will use techniques such as coding standards, unit testing, smoke testing, automated regression testing, design and code reviews etc. all of which should improve the quality of the software.

All of this testing comes at a cost. The question to be answered on every project is – how critical is this software and how much testing should we do to ensure the software is correct?

Too often the testing phase is rushed and the software goes out with an unacceptable level of defects. On the other hand, for most systems there are diminishing returns for extending the testing past a certain point. There comes a point with all software where the value of getting the software released is greater than the value gained by continuing to test for defects. This is why most commercial software gets released even though it is known to contain defects.

3. Lack of user input

For over 10 years the research company, The Standish Group, have surveyed companies about their IT projects. The No. 1 factor that caused software projects to become challenged was "Lack of User Input".

Reasons for this can include:

  • The system is being promoted by the management and so the business users have no buy-in
  • The users are too busy and have “more important” things to do
  • Relations between the user community and the I.T. team are poor

Without the involvement and input of a user representative the project is doomed to failure. This person should be a subject domain expert with the authority to make decisions and a commitment to the project timescales.

So assuming there is good user input then the challenge of translating requirements into a design begins. And this is no easy task as our next point shows.

4. Users don't know what they want until they see it

Even with good input from the users no amount of analysis of user requirements can take away an immutable fact that users only think that they know what they want. In truth, it’s not until they start seeing something, and using it, that they begin to really understand what they need. This is especially true when the software is being developed for a new idea or process that they haven’t used before.

Studies have shown that the average project experiences about a 25% change in requirements from the "requirements complete" stage to the first release. This is the famous “scope creep” problem that besets nearly all projects. It usually starts once the first designs begin to appear which cause the users to think more deeply about what they really want.

The challenge is do you a) ignore new requirements and carry on building according to the design documents and risk delivering a system that doesn’t do what the users need or b) take on changes as they arise with the result that the project expands and costs rise?

There is no simple answer to this dilemma despite the fact that various techniques, such as Agile development, have evolved to make it easier to adapt to changing requirements. Even seemingly small changes can have a big impact on a project. Each explicit requirement can lead to many more implicit requirements (by a factor of up to 50) further complicating the software.

Changing requirements during the development phase is one of the great challenges facing all software developers. It can be done but don’t ever think it’s easy. And please don’t think new requirements can be accommodated without affecting the timeline or the budget unless there is a corresponding removal of requirements.

5. There are no barriers to entry to become a programmer

There is one argument that states that software development is so hard because programming is so easy. In other words it is relatively easy to learn how to write code but there is a huge gap between that and delivering great software.

One could possibly equate it to learning a new language. Yes you may pick up the grammar and acquire a reasonable vocabulary but that’s a whole different ball game to having a fluent two-way discussion with some native speakers.

Various studies have shown that the productivity ratio between different grades of developer can be as high as 28:1. With that in mind it you surely would only want to hire the best developers. Unfortunately this is not easy as great developers are a very rare commodity.

There is no barrier to entry into the programming world and thus it is awash with many poor programmers who adversely affect projects. In addition, even potentially good young developers will still make mistakes that a more experienced developer will have learned to avoid.

It really is worth paying more for a top-class experienced developer. They will do things quicker, and better and with less code. Your project will be delivered quicker and will have fewer defects. They will save you money now and they will also save you money through the life of the system in support and maintenance costs.

6. All software is affected by external factors

Physical structures obey physical laws e.g. they are affected by gravity, mass, atmosphere etc. Through thousands of years of learning much is known about the physical world and can therefore be modelled and predicted.

Software is “mindware” and therefore doesn’t obey physical laws but it usually must conform to external constraints such as hardware, integration with other software, government regulations, legacy data formats, performance criteria, scalability etc.

Understanding and catering for all of these external factors is a near impossible task. Even a seemingly simple requirement, such as supporting multiple browsers, exponentially increases the difficulty of both building and testing software. If you then add in a requirement to support multiple versions of each browser then you are again exponentially increasing the complexity and the difficulty.

7. Estimating is an art not a science

Given the premise that all new projects are custom built, that there are no pre-built components, that the project will suffer from scope creep and that the level of skill across the development team is usually varied then it is no wonder that estimating the duration of the project can never be a scientific exercise. There are too many unknowns. If you’ve never built the same software before with the same team then how can you know how long it will take?

Of course experience guides you in your estimating and the more experience you have the more likely you will be to anticipate the unknowns. Too many projects run over because overly optimistic estimates are set by inexperienced people who expect everything to flow smoothly and who make no allowance for the unknowns.

Over the last 10 years the Agile Development method has emerged as a means of addressing these estimating issues. Whilst this is generally a more reliable way to control projects timescales, and therefore the cost, it is not suitable for all projects or even on some parts of projects.

Where projects involve complex external interfaces or new technology is being used then estimates become even harder to get right. These are risks that are often hard to quantify up-front and are usually only uncovered as the work gets done.

Summary

A software application is like an iceberg – 90% of it is not visible. The major complexity in the application lies below the waterline and is invisible to the user. The next time you are wondering why software projects appear so difficult to get right you might perhaps spare a thought for the developers. They are generally working hard to deliver on time against a tidal wave of challenges and complexity.

Comments: (7)

Dirk Kinvig - Finextra - London | 08 August, 2012, 15:33

4. Definitely.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Michael Nuciforo - Keatan - London | 08 August, 2012, 18:35

It always suprises me how many Senior managers just don't get it.  Agile definitely helps resolves some of these issues. I can't understand for the life of me how anyone can deliver using Waterfall now.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
A Finextra member | 09 August, 2012, 05:36

Paul, your list might or might not be accurate but you somehow forgot 2 top reasons that by far define the IT/software industry:

1) Client CIO's first and foremost loyalty is usually to his/her own morgage and transportability of his skills.  When the project fails the original CIO is already likely being far far away.

2) Vendor upsells what s/he has, never downsells (is it a word?) what the client needs - explicitely or implicitely.

Exceptions do happen - according to the numerous stats as much as 4% of all large IT projects are on time and within the budget.  If the client is that lucky, he should visit Makao.

Excuses, excuses, excuses...

Sincerely

Being There.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune | 09 August, 2012, 19:23

Brilliant post!

This list is self-admittedly applicable only for custom development projects. Many software projects involve customization and implementation of standard products (e.g. SAP, FLEXCUBE, etc.). In case you're planning to write separately about what makes such projects hard, here's one item to kickstart your list:

Drastic difference in what is expected from the software before implementation starts versus what is accepted after implementation ends. To cite an extreme example, many companies embarking on an ERP project expect the software to change the fabric of their company before starting the implementation but are willing to settle for accurate invoicing after ending the implementation! 

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Brian Anderson - Bloomberg - New York | 13 August, 2012, 14:31

Paul, this is one excellent compilation of why building software is so different than building manufacturing projects. It is much harder to evaluate good code and determine whether software is 'successful' at it's goal, relative to whether a bridge or tunnel is successful at allowing the safe movement of traffic. Of course, the purpose of the software project, i.e. gaming software, versus operational software for a life-saving medical device, will also determine 'how good' the software needs to be. With the 'algorithmic trading' software nightmares of recent days that threaten to take down markets and established companies, the level of quality and reliability needs to be set early on relative to the purpose. Software cowboys need to grow up and take their projects as seriously as structural engineers do when building skyscrapers. Better Quality Assurance, Testing, and extensive Alpha and Beta cycles are needed gain that first hand feedback that end users cannot possibly provide until that point in the dev cycle.

Consumers need to value important software more as well. Right now, with respect given to your number 5, anyone can hang out a shingle saying they are a 'software' development house. But not anyone can provide an ERP system, or banking system of record, or algorithmic trading platform. There is great value in software that works.

Hopefully, as the industry matures, a more pervasive understanding of the challenges you list will drive better software builds.

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Gregg Weintraub - State Street IMS - Princeton | 13 August, 2012, 18:05

Paul,

Excellent post with many good points.  But at your premise you compare software to construction.  Another notable difference in this space is that the standards construction is based upon (units of measure, materials) are significantly more standard than the base units (operating systems, hardware platforms) for computing.  This is not just a function of youth, but certainly adds complexity to software development. 

I would be interested to hear your perspective on how much this factors into the issues above and what you think should be done to address this.

Gregg

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Paul Smyth - Kynetix Technology Group - London | 04 September, 2012, 16:28

To those who commented on my article - Many thanks for your feedback.

In a short article such as this I couldn't address all aspects of software development so all the comments just add further weight to this topic.

Yes it is primarily written from a custom development point of view and reflect 30 years of experience in this area. However, I've also been involved in many CRM and other packaged software implementations and despite the fact that they are supposed to be easier to implement because it's off-the-shelf software, I can safely say that that is not usually the case.

Regardless of the type of software, custom vs packaged, I see as much confusion in the minds of users about what they really want and expect as Ketharaman points out in his comment.

Brian's comments reflect many of my own beliefs. The term "Software Engineering" is used a lot. Mistakenly I believe. I don't think software is yet as disciplined a profession as engineering. As an industry there is still a long way to go before we can call ourselves real engineers.

Gregg is absolutely right in pointing out that the base units in software development are much less defined and stable as in construction. I immedately think of an analogy that building software on any platform is like building a skyscraper on quicksand.

 

Be the first to give this comment the thumbs up 0 thumb ups! (Log in to thumb up)
Comment on this story (membership required)
Log in to receive notifications when someone posts a comment

Latest posts from Paul

7 Things Every CIO Should Know About Mobile App Development

04 December 2013  |  6536 views  |  5  |  Recommends 0 TagsMobile & onlineGroupBusiness Knowledge for IT

Why the 10,000 Hour Rule Matters in Software Development

22 November 2012  |  5107 views  |  1  |  Recommends 0 GroupBusiness Knowledge for IT

So You Want Your Software On Time and On Budget?

03 October 2012  |  4024 views  |  2  |  Recommends 0 GroupBusiness Knowledge for IT

Stop Wasting Money On Software You Don't Need!

10 September 2012  |  4289 views  |  3  |  Recommends 0 GroupBusiness Knowledge for IT

7 Reasons Why Software Development Is So Hard

08 August 2012  |  24926 views  |  6  |  Recommends 1 GroupBusiness Knowledge for IT

Paul's profile

job title CEO
location London
member since 2012
Summary profile See full profile »
CEO at Kynetix. Driving our best practice approach to software development.

Paul's expertise

What Paul reads
Paul writes about
Paul's blog archive
2013 (1)2012 (4)

Who is commenting on Paul's posts