Blog article
See all stories »

Little Big Data

Big Data is one of the trends of the moment, with beautiful infographic representations of everything from social networking, video uploads, web searching and app purchases. These are, no doubt, important tools in shaping our strategies in payments. However, this is not new.  The challenges of big data as defined by Doug Laney, the father of Big Data in 2001, are the 3 Vs - Volume, Velocity, and Variety – and his main example was e-commerce.

Well the challenge has only grown bigger, every system processing payments has grown in all of Doug Laney’s metrics. Every company now has its own ‘Big Data’ challenges – a large amount of data, with millions of transactions added daily and an increasing number of channels it is receiving it from.

The challenges are how to use this data to benefit the business. Increasingly, real-time analysis of this data becomes important – to track trends for availability, fraud and risk. These are all vital factors for ensuring a system delivers its maximum return on investment, where margins are increasingly being squeezed by competition and regulation. The challenge is focusing this massive amount of data into something that can be acted upon. Modern dashboard design certainly helps this – the key to efficiency being the right data at the right time.

The off-line analysis of this data is also key, the trends both in traditional business intelligence methodologies  and the tangential Big Data analysis are all key to ensuring profitability for the future.

The surprise is that many organisations just throw this data away. Even data warehouse data is often truncated such that information that could be used in analyses is lost forever.

So, whilst the challenges and scales of big data sets with its Terabytes of data do not seem immediately relevant to individual companies in the payments chain,  I believe they are.

The question is – what is your Little Big Data strategy?  Are you using what you have to its best benefit to the business?


Comments: (3)

A Finextra member
A Finextra member 20 September, 2013, 20:00Be the first to give this comment the thumbs up 0 likes

David, good article on Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at

A Finextra member
A Finextra member 22 September, 2013, 12:21Be the first to give this comment the thumbs up 0 likes

David, Great point you make. I totally concur. And while I appreciate the reference to the "3Vs" I defined over 12 years ago, we need to also recognize SGI scientist John Mashey as coining the term "big data" several years before that. For your readers interested in my original piece on the "Three Dimensional Data Challenge", I have unearthed it from the Gartner archives:

Cheers, Doug Laney, VP Research, Gartner, @doug_laney 

A Finextra member
A Finextra member 25 September, 2013, 07:47Be the first to give this comment the thumbs up 0 likes

Doug, Thank you for your kind words – and apologies to John – so I guess a ‘shout out’ to him too.

Interesting stuff, which I feel will start to shape us all, whatever we do.