I've been reading reviews on Destructoid for a long time now; probably since they started doing 'em. About a year ago, amidst much debate about the validity of review scores from various sites (continuing today still), Aaron Linde posted the first Destructoid Review Guide, and there was much rejoicing.

At least there was for a short while. It wasn't long until the complaints started back up again. People claimed that games were being rated too low solely to get hits, rather than reflecting on the quality of the games. So a few months ago, Jim Sterling released the Destructoid Review Guide Version 2.0, reminding us that yes, Destructoid editors use the full ten point scale, and that they don't rate games low just for controversy.

I have collected data on the past year of reviews, since Aaron posted the first review guide, up through the most recent review on Mortal Kombat vs. DC Universe, and as I will show, not only are they not rating games lower than they should, it is quite possible that even the Destructoid editors overrate games, utilizing the maligned "IGN Scale" of 5 to 10. Without further ado, I present histograms of the review scores awarded by the Destructoid staff. The first histogram (at the top of the page) shows the data between Aaron Linde's guide posting and Jim Sterling's guide posting, the second shows the data between Jim's review guide and today, and the third shows all of the data since the first review guide was posted.





The first chart shows an interesting, almost linear relation of score to occurrence between the scores of two and nine, with each subsequent score being more common than the last. It shows only two ones awarded, both of which were for Eternity's Child. It should be noted that the half-point scores generally occur less.

The second chart doesn't contain quite as many data points, so it isn't as smooth as the first. It still shows a surprising trend: eight and nine dominate the chart, each with twice as many occurrences as the next most common score. Only two games scored between zero and two (Facebreaker and Warriors Orochi 2), whereas thirty-four games scored between eight and ten.

The third chart is just the sum of the first two. With the most data points, it shows an obvious bias toward the right side of the chart, or the higher review scores.

So what causes this? Frankly, I could come up with a number of reasons. While Destructoid is large, it still isn't IGN or Gamespot, whose employees review and score every single game that is released, and so the Destructoid review team is probably more likely to play and review only the games that are likely to enjoy. It could be that in general, games that come out these days are mediocre at worst, and really awful games only come out every once in awhile. Or, the most dire possibility: perhaps some of the reviewers don't quite understand the review guides themselves, and are still grading on a five to ten scale.

Jim Sterling wrote in his review guide, "We try and review games based on what we like or know about most so if, for example, Colette rates Chocobo Cloud Smiles 2: Chocobonkers a 2 out of 10, you know it's got to be bad." While that's true, this review philosophy is potentially faulty, because if, for example, Colette rates Chocobo Cloud Smiles 2: Chocobonkers an eight out of ten, then it doesn't really tell us anything. Sure, a chocobo fanatic loved it, but what does that mean for the rest of us?

Let's step back for a moment and consider the second point though. I've given it a bit of thought at this point (in case the graphs weren't indication enough), but how should the histograms be shaped? Do we consider a score of five to be "average" or "mediocre"? If a score of five means that the game is average, then the histogram should show a nice bell curve, with four, five, and six being most common, while zero and ten are least common. If a score of five means the game is mediocre, or in other words that it is as good as it is bad, as enjoyable as it is unenjoyable, as exciting as it is boring, then it is possible for the histogram to be shaped in any way imaginable. I bring this up because depending on how one defines the full scale, there may not be anything necessarily wrong with the review scores, no matter how interesting.

Still, I wanted to more fully explore the possibility that some editors are more "at fault" for the score buffing phenomenon apparent in the total histograms. So for every editor in my data set who has written ten or more reviews, I have tabulated separate histograms. These follow, with commentary for each.



Arguably one of the most called out of the review staff for giving undeserved low scores is Reverend Anthony. His scores, however, are the most spread out of all of the editors highlighted here. Despite his negative infamy, his average score awarded is 6.71, nearly two points above an average score of five. Some points to note are that he has awarded as many threes as he has seven-point-fives and nine-point-fives. Even though his is the most spread out, it is still biased toward the high end, with only one game getting a review score between zero and two-point-five.



Brad's histogram is undoubtedly the most like a bell curve centered around five out of any of the reviewers. I've become particularly interested in his reviews after seeing these data, because it shows a lot of promise for the proper use of the one to ten scale, but it really needs more data points in it before we can draw any conclusions. One thing you may note is that he almost never awards half-point scores, which I really prefer.



And now we begin to see the more telling data of skewing toward high scores. Aside from two games (Baroque and Infinite Undiscovery), Colette has not given any games a score less than seven. Her average score awarded is 7.38, and her most common scores awarded are seven-point-five, eight, and nine.



Dale's histogram is almost identical to Colette's, with one game getting a four (Zoids Assault), and no others receiving a score less than six. His average score awarded is 7.80, with his most common scores being eight and nine.



Among all of the charts I have prepared, Dick McVengeance's (Brad Rice's) has probably the most interesting shape. It's clearly bimodal, but I'm not sure what we can conclude from that. It's as if he is actually reviewing on a binary scale; either he likes a game or he doesn't. Even so, his average score is a full point and a half above five, and his most commonly awarded score is eight-point-five.



Jim and Anthony are tied for total number of reviews within this data set at twenty-six reviews in the past year (that's one every other week on average, as it turns out), and he is another common target for accusations of doling out lower scores than were deserved. However, his is not that different from Colette's or Dale's, except that he has given a total of four games scores less than five. His histogram is still pretty heavily biased toward the right, with an average over a point and a half above five, and most common scores of seven and nine.



I'll be honest; part of the reason I was motivated to look at individual editors' scores in this analysis was because I feel that Jonathan is not critical enough of the games he reviews. The histogram really speaks for itself (but that won't stop me from pointing out various parts). The lowest score he has awarded in the past year of reviewing is a six (for Samba de Amigo), and it's the only score he has given out less than seven-point-five. His average is way up at 8.44, and his most commonly awarded (twice as many as the next most common scores) is a nine.



Lastly, we've got the histogram for Destructoid's Editor-in-Chief. It's like Anthony's in that his scores are fairly evenly distributed (aside from an overabundance of eights), but it's unlike Anthony's in that it appears Nick is using the IGN five-to-ten scale. The entire left half of the graph is blank, and his average score is 7.61, just about halfway between five and ten.

So there you have it. What do you think about this? Is the high score biasing because the reviewers review games that they are more likely to enjoy? Is it just that some people are extremely enthusiastic about games and they give everything a high score? Is it a case of the reviewers not adhering to the two posted review guides? And perhaps the most important question: is this an issue that requires addressing, or should we just go on as we have been, calibrating our judgment of scores based on the author or ignoring them altogether?

FOOTNOTE: This analysis is not meant as an attack on any of the reviewers. I love them all as people, and most of them as writers too!