Lies, Damn Lies and Small Sample Statistics.

It’s easy to get lost in the idea that the whole world is close to what you have experienced. Realistically the only thing we have to go by is what we see and hear day by day and philosophically we can’t even really prove that anything else exists outside our own sphere of influence. Before I derail this post into a lot of hand waving about cognition and awareness I wanted to explore the world of misrepresentation of data through the use of either cherry picking results or through sample bias using small or particular populations.

Cast your mind back to 2004, for the Australians among us they would remember that this was the time of the federal election, and the last time that John Howard would win his bid for Prime Minister. Back then I was still a teenager but it was the first time I was eligible to vote in the election. Speaking to all my friends and family I was convinced that this year we would oust Howard and usher in new blood to revive what I saw to be a stagnant government. You can then imagine my shock as not only did the Liberal party win, but did so by taking 5 seats away from Labor. The politically inclined among you would realise that typically Canberra is a Labor electorate and if took nothing but opinions from the people within Canberra you would come to the same conclusion. This was classic sample bias and it led me to become more involved in politics, as I now knew that I couldn’t trust just the people I talk to in order to extrapolate to Australia as a whole.

Just today a good friend of mine sent me this article that also used flawed logic and small sample size to make wild accusations about the general health of the gamer population:

ADULTS who play video games may suffer higher levels of depression and weigh more than non-gamers, according to a study released today.

The study, conducted by the US Centres for Disease Control and Prevention (CDC), Emory University and Andrews University, found “measurable correlations between video-game playing and health risks”.

The study – “Health-Risk Correlates of Video-Game Playing Among Adults” – is being published in the October issue of the American Journal of Preventive Medicine.

The researchers surveyed 562 adults ranging in age from 19 to 90 in the Seattle-Tacoma area of Washington state. A total of 45.1 per cent of those surveyed reported playing video games.

The sample is, to say the least, incredibly biased. Let me just pick out a couple of the problems with the data set they have used:

• The sample size is incredibly small to be able to draw any substantial conclusions.
• The use of weasel words like “may” and “measurable correlations” are not something you find in well researched scientific reports.
• The sample is taken from one area, which according to this lovely animated graphic from the Center for Disease Control and Prevention shows that they have an obesity rate of greater than 30%.
• Repeat after me, correlation does not equal causation.

If we were say to apply this to my group of friends (of which the sample size is approximately equal in gamer/non-gamer distribution with a fifth of the size) you would probably find that gaming has little to no correlation to obesity and depression. In fact you’d see that gamers on average tend to be healthier, but the problem is that whilst we all identify as game players we each have our own reasons for keeping fit and healthy. The numbers used and conclusions drawn are misleading at best and anyone who’s spent even a small amount of time working with statistics will tell you that using a sample size of 0.0094% (564 people divided by 6 billion in the world) of the population can not be relied on.

Statistics are the one thing that everyone is familiar with but no one seems to understand completely. All too often I’m seeing reports being made or news articles being published that use fatally flawed mathematics and unfortunately this often misleads people to believe things they otherwise would not. For the mathematically inclined among us it then becomes a battle of education to give people the tools so they can break down the arguments analytically, however there’s only so far you can go before people stop listening.

As usual there’s a slight anti mass media bias to this post but what I truly desire is for people to question information that is given to them. We humans are wired to turn off our sceptical parts of our brain when an expert tells us something and this is why we need to build up our bullshit detectors so we don’t get fooled by the people who wield the power of statistics. It just so happens that the biggest abusers of this power are the media.

And yes the irony of using statistics to disprove statistics isn’t lost on me. I’ll still take the moral high ground on this issue however 🙂