This week Chad Concelmo wrote an editorial explaining review scores, and how the scores could not be used as a comparison system. I'm not sure why Concelmo, who admits he rarely does reviews to write about such a topic, but I love Concelmo's Memory Card features, and I think he has an understanding of the narrative in games that is uncommon. However, this article is silly, for two glaring reasons.
First, Chad explains the process of tallying, and that it's basically consisted of a percentage of how much the game lives up to it's potential:
"Each review and its respective numerical score (between 1 and 10) is based on the quality of that particular game. If a game gets a 5.0, it is completely average. More specifically (and this is super important!), giving a game a 5.0 means that the game being reviewed could, technically, be about five points better if certain things were improved. In more mathematical terms, this means the game is living up to 50% of its potential."
He goes on explaining that if a game is scored a 9 it only needs one element to improve for the game to live up to it's potential, and a 5 means it needs to improve on more things to garner a better score. This doesn't necessarily contribute to his argument, because saying this asserts that a numerical score is a quantification of the quality of the game. Unfortunately the math is anything but simple.
Now, according to Concelmo's perspective, a DS dungeon crawler that receives a 9 is not necessarily better than a PSP puzzle game that got a 6, yet his admission of the scores correlating with the game's overall quality suggest that a game that is rated a 9 is more likely to be well received on a general basis, not on an individual level. It doesn't really make any sense if a game that needs less to improve is just as good as a game that needs more.
Everybody already knows that there is no completely objective way to compare games, but what's the point of a score if it can't even give us an approximation of how good a game is, when a '6' game could subjectively be as good as a '9' one?
Second, If Concelmo's assertion is right, it hugely fucks up the integrity of the scores if I can't tell if Metal Gear Solid 4 is better than Lego Rock Band based on the review scores. Additionally, if reviews can't consistently be compared (even approximately), why implement a numerical score review at all? Don't implement a numerical score and chastise people for using it to make comparisons, if it really is supposed to be apples to oranges or brussel sprouts to cupcakes.
This also brings me to another point why Destructoid's review system is flawed; there HAS to be a numerical score system because the review HAS to be aggregated. This is dumb because it not only is the numerical system incredibly inconsistent already (if Concelmo is right), it suggests that users have to venture to other websites to get the whole picture. I understand that being in Metacritic helps the industry take you 'seriously', but how seriously can they take you when there isn't any way to compare scores?
Anyway, I love a lot of the articles and I read Dtoid every day, but after reading Chad's article I think that the review scoring system could use a makeover to elimnate confusion and the need for complicated clarifications. I apologize for kicking off my blog with a negative note, but I felt like I had to say something. Please comment!