According to Gartner, Hadoop investment remains tentative in the face of sizable challenges around business value and skills, with a lack of training and knowledge about the free, open source Java-based programming framework inhibiting growth. 

The survey, which was conducted in February and March 2015 among 284 Gartner Research Circle members comprising IT and business leaders from various sectors, found that only 125 respondents had already invested in Hadoop or had plans to do so within the next two years.

"Despite considerable hype and reported successes for early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next two years," said Nick Heudecker, research director at Gartner.

"Furthermore, the early adopters don't appear to be championing for substantial Hadoop adoption over the next 24 months: in fact, there are fewer who plan to begin in the next two years than already have."

Only 26% of respondents claim to be deploying, piloting or experimenting with Hadoop, while 11% plan to invest within 12 months and only 7% are planning investment in 24 months.

Responses pointed to two interesting reasons for the lack of intent. First, several responded that Hadoop was simply not a priority. The second was that Hadoop was overkill for the problems the business faced, implying the opportunity costs of implementing Hadoop were too high relative to the expected benefit.

Figuring out how to get value from Hadoop was cited by 49% of respondents as a key concern, as was demonstrating the value. The absence of skills has long been a key blocker too, with 57% of respondents citing the skills gap as a major adoption inhibitor.

Technology vendors claim their products can address the skills gap, but Gartner says that while tools are improving they primarily support existing highly skilled users – rather than elevate the skills already available in most organisations.

Hadoop vendors are responding to this challenge by offering a variety of training options. However, Gartner estimates it will take two to three years for the skills challenge to be addressed.

"With such a large incidence of organisations with no plans to adopt future demand for Hadoop looks fairly anaemic over at least the next two years,” said Merv Adrian, research vice president at Gartner. “Moreover, the lack of near-term plans for Hadoop suggest that - despite continuing enthusiasm for the big data phenomenon - demand for Hadoop specifically is not accelerating."

"The best hope for revenue growth for providers would appear to be in moving to larger deployments within their existing customer base."

Hadoop background
Hadoop is a free, open source Java-based programming framework, overseen by the Apache Foundation, which supports the processing of large data sets in a distributed computing environment. It increases resiliency, data speeds, and is a key enabler of technology vendor’s big data analytical solutions in the search, advertising and unstructured data arena.

Adoption by banks, retailers and other end use clients in speeding up customer monitoring behaviour, social media, processing and other unstructured end uses is still not universal at this stage.

The Hadoop programming framework is so named because the creator, Doug Cutting, named the open source Java-based framework after his child's stuffed toy elephant, hence the elephant logo, it uses. 

Muliple user bar & skills shortage inhibit growth 
"Early Hadoop projects typically involve a small number of users and this no doubt keeps user populations down at this stage of the market. Moreover, the Hadoop stack remains unsuitable for simultaneous use by multiple users, also keeping numbers down,” concluded Heudecker, while also citing the skills shortage for the slow uptake.

"One of the core value propositions of Hadoop is that it is a lower cost option to traditional information infrastructure approaches. However, the low numbers of users relative to the cost of cluster hardware, as well as any software support costs, may mean Hadoop is failing to live up to this promise."

The tiny number of survey respondents that are actively piloting or deploying Hadoop at the moment told Gartner that only small numbers of their internal users are actually accessing the cluster with 70% of early adopters reporting only between one and 20 users accessing Hadoop. A surprising 4% of early adopters reported zero users.