I've often mulled over this question, bringing it up to my friends, trying to find an answer. The latest response I was given, "A game that has character progression", e.g. a game where you get stronger with the use of experience points, armor, weapons, etc. I took that answer at face value and went along my merry way, until I gave it some thought and realized that it was complete bullshit. There are tons of games were you upgrade your character with new armors and weapons that aren't considered RPGs.
All of these games have gear based character progression. The only difference between these games and others that classify themselves as RPGs is the lack of experience points and "leveling". Is that the only thing that makes a RPG what it is? leveling? Of course not, since there are games that are not RPGs and also include experience points, like Dead Rising for instance.
So what is it that makes a RPG?
Story? Clearly, this doesn't make a RPG since there are many, upon many games such as Bioshock or the Legacy of Kain series that tell a compelling story without being classified as a RPG.
Exploration? If this is what makes a RPG then you would have to discount Final Fantasy, the largest RPG franchise there is, since there is little to no true exploration in any of the FF games.
Combat Mechanics? These days there are two RPG camps, Western RPG and Japanese RPG, and there is a clear difference between the two types; namely real-time combat vs. turn-based. To me, the fact that there is this conflict signifies that neither one nor the other is essential to the classification of an RPG. Either one will do.
Choice and Consequence? Though these elements are pretty much none existent in JRPGs, WRPGs revel in the mechanic of making a character's choices affect your game. Personally, I think this to be the most obvious answer to what makes an RPG, since "role-playing" is about taking a role and doing what you will with it. The obvious revelation of mine quickly crumbled away as thoughts of Fahrenheit and Heavy Rain came to mind. Games created solely on the principle of choice and consequence. Are these games RPGs? Are they more RPG than other games that claim to be RPGs? My mind reels.
So, what is a RPG? Is there any one thing, one true and pure staple of what RPGs should be? Is the sum greater than the equal of it's parts? You tell me, I don't know anymore...
Planescape:Torment might not have anything particularly unique about it's gameplay, but the characters, world, and story are what really makes this game stand out.
In the days of old (1999), when Black Isle Studios ruled the land of RPGs, creating such acclaimed games as Fallout (1997), Fallout 2 (1998), and Icewind Dale (2000), a game was created featuring a world so stupendously unique that it has yet to be revisited with the media of video games. That world was Planescape (which is actually a pen and paper rpg, but we're talking video games here). In this world, a door could be anywhere (a hole in the ground, an archway, or even in a dresser), a key could be anything (a piece of junk, a coin, a knife), and could lead anywhere in the known, and unknown, multiverse.
I mean really, the main character was immortal. How often do you get to be immortal these days? (Lost Odyssey) Hmmm, ok, how often do you get to be a cool immortal? (Kaim was kinda cool) Fine, how often do you get to be an immortal that looks like he's been carved up more than the Christmas goose; that wakes up in a morgue with no memory of who he is; that has a talking skull as a companion; that gets to make out with a tiefling; that gets a githzerai with a sword made out of chaos; that gets a celibate succubus priest on his team; that gets to journey to hell and back, literally, just to find out who the hell he is and why he's immortal? (You wake up in a morgue without a memory in Shadowrun) ...fuck you.
I guess the point I'm getting at is that the world of Planescape should not be forgotten by the gaming and game developing community. That world has infinite potential, and the fact that there is only one videogame using it seems kinda sad to me.
I'll leave you with some words said to me by the night hag Ravel
"What can change the nature of a man?"
(What's a night hag)