Blog article
See all stories »

Why Is Software Still Built From Scratch?

A few months ago, I'd visited the South Indian temple town of Tirupati with my family. On the way, whenever the train stopped at a station, the engine was switched off. But the air-conditioning still worked because it was powered by huge battery packs installed in the AC coach. Coincidentally, the manufacturer of these battery packs is located close to Tirupati. I know this company from the time it purchased my employer's ERP in the early 2000s. Through the years, I've stayed in touch with the ex-CEO of the unit. 

I met him during my latest trip and introduced him to my family as the guy who "built the electronic circuit that manages the battery packs on AC coaches of Indian Railways trains".

He corrected me, saying, "I assembled the circuit using available integrated circuits without building every part of it."

Yes, indeed. Since times immemorial, automobiles, printed circuit boards and many other products are assembled from standard components, which can be sourced from catalogs.

But not software.

I joined the software industry two decades ago. From then to today, I've been hearing of technologies like SOA, CBSA and microservices that purportedly allow programmers to build systems by assembling prebuilt functionality.

But that vision has still not turned into reality.

Once upon a time, developers had to build software systems from scratch perhaps because there was no prebuilt functionality.

Then, it's quite likely that developers built reusable components by using technologies like SOA. But, despite the existence of methodologies like CBSA, development teams still had to build software from scratch because they didn't know about the existence of reusable components.

Then came innovations like open source repositories and open API marketplaces.

For the uninitiated, an open source repository (e.g GitHub) contains prebuilt source code for components that can be used by programmers to develop their own systems; and an open API marketplace (e.g. Algorithmia) expose readymade functionality that can be accessed by developers from their systems via application programming interfaces.

In theory, if you want to develop a new system, you should be able to review the catalog on these platforms, pick and choose the component or API you need, and integrate the prebuilt functionality into your new system.

In practice, this has worked reasonably well in building software pilots / proofs of concepts / MVPs.

However, when it comes to enterprise systems, these platforms haven't made a big dent in the traditional way of building software systems from scratch. I'll outline the key hurdles of changing the time-worn development methodology by continuing to use the PCB analogy:

  1. A 12μF (twelve microfarad) capacitor will work on any PCB. Whereas, since software is heterogeneous, a .NET component won't work in a Java system.
  2. You can't manufacture a capacitor ab initio even if you wanted to - at least not within the timescales of most PCB development projects. Whereas, most development teams believe that they can develop any piece of software from the ground-up within software project timescales.
  3. You use a capacitor as it is. Whereas developers inevitably need to make some changes to borrowed code before being able to integrate them into their systems. According to open source foundation rules, you're required to share the enhanced / modified code back with the community. That can be tricky if it contains a company's secret sauce. You either flout OSF rules - à la  the investment bank described in Michael Lewis's book Flash Boys - or eschew open source code altogether.
  4. You can buy a capacitor. But you can only license an API, which is not the same as buying it. The license is subject to the Terms of Service of the API owner ("OEM"). Now, many of these TOS are ambiguous and easy to infringe without any wrongful intent. The deadpool is littered with startups that got on the wrong side of the TOS of an 800-pound gorilla's API.
  5. Once you plug a capacitor into your PCB, it will work the same way throughout its lifetime. Whereas an API won't. When its OEM releases its next version, you'll need to modify your software such that it works with the updated API. We faced a costly change to our HEATMAP360 social intelligence platform when Twitter changed its API specs.
  6. If the manufacturer of a capacitor (also "OEM") folds up, your PCB will still work. But, if the API's OEM dies, your system will stop working and you'll need to find an alternative source for the said functionality. As illustrated by the examples in Does Cloud Increase Vendor Risk? (hyperlink removed to comply with Finextra Community Rules but this post should appear on top of Google Search results when searched by its title), the death of a few API OEMs has caused existential crises for many software systems and their owners.
  7. Then there's the big elephant in the room that no one talks about: Programmers don't like working on code written by anyone other than themselves.

As a result of these challenges, it has been hard to develop enterprise-grade software systems by assembling prebuilt functionality.

That's not to say it can't be done.

But cracking the Holy Grail of software engineering will require structural changes in the way software is packaged and sold. The purchase process for software should mimic that of electronic components whereby customers can:

  • discover software components in an open catalog
  • find out their specifications and select which components they need to assemble their system with
  • buy the selected components outright without onerous licensing terms or dependency upon any form of first- or third-party IP
  • be assured of getting the same functionality throughout the life of the software, and
  • modify components without having to share the source code with external entities.

If you're wondering "what's in this" for a software product vendor, you're not alone. Actually, this model is more closely aligned with IT services. More on that in a follow-on post.


Comments: (4)

A Finextra member
A Finextra member 16 May, 2017, 12:51Be the first to give this comment the thumbs up 0 likes

Is it in a Software Companies interest to build software you can reuse indefinitely - or charge you to build bespoke each time. This is why ACI and CSC get away with charging huge fees for customization of the application (as only 70% of a COTS Product meets a customer's requirements - broadly speaking). Some software is sold as pre-integrated with other packages (at an Interface-level) – such as a Fraud Detection System (FICO Falcon); Credit Risk System (FICO TRIAD) and Card Management System (normally VisionPLUS or CAMS II ODS). 

Chandrashekar Rao Kuthyar
Chandrashekar Rao Kuthyar - SangamOne Connected Services Pvt Ltd - Bengaluru 17 May, 2017, 03:02Be the first to give this comment the thumbs up 0 likes

<CS> Compliments to the author for a thought-provoking article. The comparison with the capacitor is interesting. If I buy a capacitor from Philips, it will work the same way throughout its life. I will not be taken by surprise when some internal change happens within Philips.   The idea of catalog of software components available openly in the market is good</CS>

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 01 June, 2017, 16:27Be the first to give this comment the thumbs up 0 likes


Apologies for missing out acknowledging and replying to your comment earlier. 

Yes, as I'd hinted at the end of my post, it's not in the interest of traditional product vendors to help customers to assemble software systems from prebuilt components. The operating model required for that may be more appealing to services vendors.

That said, I recently stumbled on to a couple of product vendors who highlight reusability e.g. Avoka:  

"The Avoka Transact™ design tool provides pre-built components for rapid form creation as well as the ability to customize the experience."

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 01 June, 2017, 16:35Be the first to give this comment the thumbs up 0 likes


Apologies for delay in responding to your comment.

TY for your kind words.

Yes I agree. In fact my point #5 resonates with your comment about capacitor.

PS: I Googled for <CS> tag, couldn't find anything on page 1 of SERP, didn't bother to go beyond that. I hope it's not a variant of </sarc> tag or something like that:)

Now hiring