16 July 2018

44975

Retired Member

3,614Posts 14,165,234Views 4,515Comments
Finextra community

Data Management 101

A community blog about data and how to manage it

When Big Data is not Big Understanding

08 April 2014  |  3649 views  |  0

Good article from Tim Harford (he of the enjoyable "Undercover Economist" books) in the FT last week called "Big data: are we making a big mistake". Tim injects some healthy realism into the hype of Big Data without dismissing its importance and potential benefits. The article talks about the four claims often made when talking about Big Data:

  1. Data analysis often produces uncannily accurate results
  2. Make statistical samplying obsolete by capturing all the data
  3. Statistical correlation is all you need - no need to understand causation
  4. Enough data means that scientific or statistical models aren't needed

Now models can have their own problems, but I can see where he is coming from, for instance 3. and 4. above seem to be in direct contradiction. I particularly like the comment later in the article that "causality won't be discarded, but it is being knocked off its pedestal as the primary fountain of meaning."

Also I liked the definition by one of the academics mentioned of a big data set being one where "N = All", and that you have "all" the data is an incorrect assumption behind some Big Data analysis put forward. Large data sets can mean that sample error is low, but sample bias is still a potentially big problem - for example everyone on Twitter is probably not representative of the population of the human race in general.

So I will now press save on this blog post and help re-enforce the impression that Big Data is a hot topic...which it is, but not for everyone I guess is the point.

 

TagsRisk & regulationWholesale banking

Comments: (0)

Comment on this story (membership required)

Retired's profile

job title
location
member since 2014
Summary profile See full profile »

Retired's expertise

Member since 2009
3595 posts4,515 comments
What Retired reads

Who's commenting on Retired's posts

Pooja Golakonda
Behzod Sabirov
Ketharaman Swaminathan
Melvin Haskins
James Treacher
Kenneth Marritt
Mark Santall
Alexander De Lange
Graham Seel
Kishore Meda
Willem Lambrechts