20 July 2018

Dave Lock

David Lock - Insider Technologies

12Posts 33,541Views 7Comments

Little Big Data

20 September 2013  |  4405 views  |  2

Big Data is one of the trends of the moment, with beautiful infographic representations of everything from social networking, video uploads, web searching and app purchases. These are, no doubt, important tools in shaping our strategies in payments. However, this is not new.  The challenges of big data as defined by Doug Laney, the father of Big Data in 2001, are the 3 Vs - Volume, Velocity, and Variety – and his main example was e-commerce.

Well the challenge has only grown bigger, every system processing payments has grown in all of Doug Laney’s metrics. Every company now has its own ‘Big Data’ challenges – a large amount of data, with millions of transactions added daily and an increasing number of channels it is receiving it from.

The challenges are how to use this data to benefit the business. Increasingly, real-time analysis of this data becomes important – to track trends for availability, fraud and risk. These are all vital factors for ensuring a system delivers its maximum return on investment, where margins are increasingly being squeezed by competition and regulation. The challenge is focusing this massive amount of data into something that can be acted upon. Modern dashboard design certainly helps this – the key to efficiency being the right data at the right time.

The off-line analysis of this data is also key, the trends both in traditional business intelligence methodologies  and the tangential Big Data analysis are all key to ensuring profitability for the future.

The surprise is that many organisations just throw this data away. Even data warehouse data is often truncated such that information that could be used in analyses is lost forever.

So, whilst the challenges and scales of big data sets with its Terabytes of data do not seem immediately relevant to individual companies in the payments chain,  I believe they are.

The question is – what is your Little Big Data strategy?  Are you using what you have to its best benefit to the business?


Comments: (3)

Azana Baksh
Azana Baksh - LEXISNEXIS - Boca Raton 20 September, 2013, 20:00

David, good article on Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at http://hpccsystems.com

Be the first to give this comment the thumbs up 0 thumb ups!
A Finextra member
A Finextra member 22 September, 2013, 12:21

David, Great point you make. I totally concur. And while I appreciate the reference to the "3Vs" I defined over 12 years ago, we need to also recognize SGI scientist John Mashey as coining the term "big data" several years before that. For your readers interested in my original piece on the "Three Dimensional Data Challenge", I have unearthed it from the Gartner archives: http://goo.gl/wH3qG.

Cheers, Doug Laney, VP Research, Gartner, @doug_laney 

Be the first to give this comment the thumbs up 0 thumb ups!
David Lock
David Lock - Insider Technologies - Manchester 25 September, 2013, 07:47

Doug, Thank you for your kind words – and apologies to John – so I guess a ‘shout out’ to him too.

Interesting stuff, which I feel will start to shape us all, whatever we do.


Be the first to give this comment the thumbs up 0 thumb ups!
Comment on this story (membership required)

Latest posts from David

There's one thing worse than being talked about.

07 April 2014  |  2016 views  |  0 comments | recomends Recommends 0 TagsInnovation

Data hoarding - or Big Data

01 April 2014  |  1404 views  |  0 comments | recomends Recommends 0 TagsPaymentsRisk & regulation

Prepaid cards: a maturing market?

20 November 2013  |  3319 views  |  0 comments | recomends Recommends 0 TagsCardsPayments

Little Big Data

20 September 2013  |  4405 views  |  2 comments | recomends Recommends 0 TagsPaymentsInnovation


05 July 2013  |  3536 views  |  2 comments | recomends Recommends 0 TagsPaymentsInnovation

David's profile

job title Business Solutions Manager
location Manchester
member since 2013
Summary profile See full profile »
Providing solutions and support with Insider Technologies Monitoring solutions

David's expertise

Member since 2013
12 posts7 comments
What David reads
David's blog archive
2014 (2)2013 (10)

Who's commenting on David's posts