Blog article
See all stories »

Demystifying The Ubiquitous Sample Size Of 2000

Back in the day, we learned in statistics that you need a sample size of at least 2% of the size of population to make statistically significant conclusions about the behavior of the population. In common speak, the expression "statistically significant" means "valid".

Nevertheless, if you're like me, you regularly come across surveys on populations of hundreds of millions of members that use sample sizes of 2000, which is way less than 2%. A few examples:

EXAMPLE 1

(https://twitter.com/pennycrosman/status/1075440678363107329)

EXAMPLE 2

Not even 2K...

(https://twitter.com/EuropeElects/status/1086794905580695558)

EXAMPLE 3

An Amazon Checking Account Could Displace $100 Billion In Bank Deposits (But It Won't)

EXAMPLE 4

Most Americans foresee death of cash in their lifetimes

EXAMPLE 5

Barely three percent of the 2000+ consumers surveyed by the FCA had made an investment in cryoptoassets such as bitcoin and ether

----------

For reference, population of Great Britain is ~60 million and of USA is ~300 million.

The sample sizes in these studies work out to 0.0033 to 0.00066 percent of the respective population, which are well short of 2%.

Should we debunk the findings of such studies?

At one time, I thought yes.

But, now, I'm not so sure. Many of these studies were published by well-reputed media outlets that can't be dismissed so easily.

So, I decided to probe the topic further.

----------

I came across this online sample size calculator, which says a sample size of 1006 yields a 95% confidence value of results with 3% error margin for a population of 300 million.

Not convinced with the above, I found a formula to calculate error margin here. When I plugged in the values, I found the results tallying with the above.

I was also intrigued by the following line on the sample size calculator website:

“The sample size doesn't change much for populations larger than 20,000.”

----------

What gives?

I suspect it has something to do with the composition of samples and populations.

Surveys make a tacit assumption that their samples are "truly representative" of the population. To establish that, we can be guided by the following maxim that I call the Gallup Soup Principle:

Some populations are homogeneous i.e. of the same nature. For example, liquids that have been given a good stir. (Unlike James Bond's Martinis, which are shaken, not stirred!).

Other populations are heterogeneous i.e. diverse in character. For example, the terrain of the earth.

A nation tends to be homogeneous on some parameters (e.g. nationality) but heterogeneous on some others (e.g. income).

For the sake of this post, homogeneous refers to "homogeneous by nature" as well as "becomes homogeneous after being given a good stir"; and heterogeneous means "heterogeneous by nature" and "remains heterogeneous even after being given a good stir". Unlike liquids, you can't stir many populations, so "good stir" effectively happens by taking a random sample of those populations.

For homogeneous populations, a sample size of mere 2K will exhibit the characteristics of the aforementioned spoonful of soup. Accordingly, results of surveys conducted with such a small sample could be valid for the entire population, however large it is.

For heterogeneous populations, a sample size of 2K is unlike the spoonful of soup. Accordingly, results of surveys conducted with such a sample may not be valid for the population. In fact, such surveys often yield misleading or contradictory results.

Misleading results

  • Indians don't speak Hindi (Survey of 2000 Indians in Tamil Nadu, a southern state of India in which Tamil is the local language)
  • America is all planes and has no mountains (Survey of the terrain of 2000 square miles of Kansas)
  • 95% banks have innovation labs (Survey of 100% of banks who have innovation labs)

Contradictory results

  • Cash is dead v. Cash in circulation is growing
  • Branch is dead v. Banks are opening new branches
  • Omnichannel shopping is BS v. Book Online & Collect At Store is the future of retail, and so on.

----------

Whether a survey with a mere 2K sample size will deliver valid results for a population of millions depends on how homogeneous or heterogeneous the population is.

2751

Comments: (2)

Melvin Haskins
Melvin Haskins - Haston International Limited - 05 May, 2020, 08:57Be the first to give this comment the thumbs up 0 likes

Your very last paragraph is the most important. Whilst the USA has a population of 330 million in 50 states, there is no comparison between the people of New York and of Wyoming. They may be in the same country but their values, habits and lifestyles will be very different. Any survey of 2,000 in the USA has no chance of reflecting the diversity in the country.

Ketharaman Swaminathan
Ketharaman Swaminathan - GTM360 Marketing Solutions - Pune 05 May, 2020, 13:28Be the first to give this comment the thumbs up 0 likes

@Melvin Haskins: 

Thank you for your comment. 

As I pointed out in the original post, "A nation tends to be homogeneous on some parameters (e.g. nationality) ..."

A survey of 2K people each in New York and Wyoming will show that a majority of them are American citizens, with the minority comprising tourists, immigrants on H1B, F1, Green Card but not US passports. Regardless of the diversity of population between these two states, this finding will be correct for both states. 

OTOH, someone can counter that by saying, we really don't need a survey - of whatever size - to reach that motherhood conclusion. 

So, in addition to the last paragraph, one more key thing that determines the validity of survey findings based on a particular sample size is, what is the characteristic being surveyed and whether it's deep enough to require a survey at all. 

IMO, the other important factor is, is there a rigorous quantitative measure of representativeness of sample and homogeniety of population. I've reached out to at least 10 people, including a couple of people with degrees in Data Science, quizzing them on this point. I haven't received a single reply.  

Intrigued, I looked up my 30 year old MBA text book on quant methods. It says this subject lies at the heart of the objectivist-subjectivist controversy and no definitive answer can be given on it. I didn't bother to go back to my 35 year old philosophy (Humanities elective during Engineering, if you want to know!) text book to figure that one out.  

Someday, I hope I can provide a more definitive perspective on this subject but, as of now, my blog post has to end on a note of suspense!