I have a sorted history with metrics. As any blogger will attest to one of your biggest motivators is the number of hits you generate on your blog per day as each of them is (hopefully) a real person wanting to read your content. It’s quite hard to write a post every day if you’re not sure anyone will read it and for a while that was the case with this blog to. I’m lucky enough to attract about 40ish people per day and that’s more than enough for me to lose an hour¹ writing a thousand words on something. I know most of them don’t read it on the same day when I publish it but that’s just the way it is and it’s always nice to see someone trawl through a week’s backlog of my posts. It lets me know they’re interested.
Still after a while the metrics start to lose their meaning after a while and it’s usually a combination of two things. The first is you begin to realize they aren’t everything as all that should matter is that you’re doing something you enjoy (and I will attest that some people just like seeing the numbers, hey I play World of Warcraft to remember). The second, and probably most important one, is that many of the numbers can be gamed in all sorts of ways which dilutes their real meaning. Whilst its really up to the providers of the metrics to police such things it’s also the responsibility of the users of that metric to ensure they’re not being taken for a ride by their slightly more savvy counterparts.
Take for instance this post on TechCrunch which is ultimately responsible for this post:
Yahoo and Bing/MSN each added approximately 60 bps and 30 bps to 18.3% and 12.1%, respectively. Google is down, claims comScore, declining approximately 70 bps for the second consecutive month to 63.7%.
But that’s not the whole story, and investors need to caution when interpreting the data as presented by comScore, say analysts.
…
We’ve detailed how Yahoo has boosted its search market share with these ‘tricks’ last month.
When adjusted, backing out Yahoo and Bing/MSN’s use of contextual shortcuts and image slide-shows from both May and April, Broadpoint.AmTech estimates that Yahoo’s share actually declined roughly 30 bps month over month in May to 16.6%, while Microsoft Sites’ share was flat at approximately 10.8%.
Google, after a small data collection adjustment to the April data (namely a change in how Google handles searches with typos), appears to have gained roughly 30 bps of share in May to 66.4%, says Broadpoint.AmTech. However, Google’s domestic core search market share was 63.7% in May, down slightly from 64.4% in April, J.P. Morgan claims.
Whilst I’m sure that most of the “gaming” here is really just innocent changes on the behalf of the relevant search providers it does illustrate my point aptly. Metrics are great for getting a feel about certain aspects of a subject but due to their unfortunately algorithmic nature they are always susceptible to outside influence gaming certain variables to act in their favor. Just to prove how fragile some of these things can be give a quick Google of the terms “Alexa rank script” to see how quickly these metrics can turn from insightful tidbits of data to digital red herrings.
This isn’t limited to the world of hard numbers either. Take for instance humble soft drink, often one of the first things to pointed out as a major cause of health problems in the developed world. Common wisdom suggests that sugar is to blame for most of the problems and so many of the companies sought to replace it with artificial sweeteners (Coca-cola being the longest hold out on that point). This is despite the fact that the jury is still out on whether or not it’s the healthier alternative but I’d rather take my chances with good old fashioned sugar with it’s mostly known health consequences. The point here is that companies will take any metric you give them and find a way to undermine it if it will help their bottom line.
Perhaps the most bandied about metric recently is the one of market capitalization. In essence this metric is the number of shares that have been bought and paid for by investors and are not currently being traded which is then multiplied by their current value on the stock exchange. Taking that number into consideration it recently came to pass that Apple was bigger than Microsoft. If you’d care to take a glance at each of the companies in real terms however you’d know that Microsoft stands as the giant over Apple any day of the week with almost $16 billion more in revenue and well over double the head count of employees. I’m not saying that Apple isn’t a highly successful company, far from it, more the fact that the market cap metric isn’t a good metric for judging a company on its own. I’d hate to see what would happen to Apple’s market cap if say 5~10% of their shareholders, you know, tried to sell their shares.
It all takes me back to those maths classes at university. You see for 3 years I did all sorts of mathematics that all my teachers told us had untold applications in the real world. Unfortunately though most of them failed to actually show us any practical applications, leaving us with countless algorithms that were set to gather dust in the back of minds. However in the numerical analysis class our teacher finally got us to understand all the equations that we’d been learning and how to make sense of the numbers that came out the other end of an equation. Since then I’ve looked upon almost all metrics with a skeptical eye, especially those who decline to discuss how they were derived. I’d encourage you to do the same as you’d be surprised how often crazy metrics pop up.
¹Today however I lost a lot more than that because my web server, in its infinite wisdom, decided to trash this web site yet again. Whilst I did make sure to copy the blog post out of the editor before it went kaput I accidentally copied something else I needed to try and get the bloody thing running again. I would chalk it up to it just being flaky but the 26 other sites I have running on here suffered no such troubles, sigh.
