Those are the three pillars of RPGs today. Like a dark, greedy soul, these three mechanics are deep at the core of the game experience. It wouldn't be an RPG, if we wouldn't be playing a character, and progress this character through the game (with the help of XP), while choosing how the game's world reacts to the character--to a good or a bad person, at its most basic.
Unless it's a linear game. Then it doesn't matter how the world reacts: It's dictated by the story. But there are only few purely linear RPGs anymore (linear in this instance meaning that there is only one possible way to solve a problem posed by the game). This sort of storytelling, obviously, doesn't concern me at the moment (and is a whole 'nother can of worms, too).
No game designer can go wrong when doing an RPG the tried way. It's what we, the players, are used to, can instantly recognize and use, without the game having to bother to explain this to us, and thus enjoy the game immediately. This is, in general, a Good Thing.
Much like when visiting the fast food joint of your choice, you won't have to deal with surprises. But, much like visiting the fast food joint of your choice every day, this can get really old, really fast.
It has to go.
How it is today
You start a new game. You create your character, and go forth, rescuing the princess/slaying the dragon/marrying the king. On the way there, wacky hijinks ensue, providing many a tale to teach children mores.
Character
A new player doesn't know what they are doing. What's the difference between a melee and a ranged fighter in this game? What will give me the most fun? What if I change my mind about the class I want to play? And what do all these numbers on my character screen mean?
So many questions right from the start, and so little guidance.
Karma
On the heroic travels, the player is confronted with options how to react to the game world. From a killing spree, to displays of cunning linguistic skills, the player has to make decisions. These decisions are, usually, threefold: Good, neutral, or evil.
Save the children from the dragon, get Good Karma. Kill just the dragon, get No Karma. Feed the children to the dragon, get Evil Karma. This is, of course, humbug. Good and Evil aren't clear cut (unless the game world is black and white, of course), and motivation is important to evaluate ethical behavior. There's a difference if the children are rescued because mom asked the hero to, or because the evil corporate executive ordered the freed.
XP
After success in the quest, the player is rewarded. With money, phat loot, and most importantly: experience points, simulating the characters growth. Rare is the case that experience points function as currency (Off the top of my head, only the two Vampire: The Masquerade computer games, and the SNES Shadowrun game allow buying skill points any time in the game, as well as Fable II more recently). Most of the time, experience is accumulated until a certain point is reached, and *boom* the player's character gained a level in badass.
This is how most any RPG functions. Therefore, it is boring. There is no surprise in the mechanics to the veteran player, and too much surprise for the new player. And not of the good kind, the suspense creating or relieving kind of surprise, either.
How it could be done
So, how can a game go about these things instead? How can it go boldly where no game has gone before? One possibility is hiding all the number crunching. Making character progression natural. And rework the Karma Meter completely.
Character
Each and every game has a tutorial, teaching the player how the game works. Some obvious, some not so much, but ideally a part of the story itself.
And while the player learns about the game, why not let the game learn about the player?
Present the player with choices, and record how he reacts to them, and how he solves problems. If the player goes all out, and kills everything in sight, talks to NPCs about how buff the new training program made him, let the character become a broad-shouldered, gun-toting war hero.
Does the player sneak around a lot, and play with knifes when ever possible? Make the character a stealthy knife-juggling special operative.
Does the player not fight at all, instead relying on smarts and talking to solve problem? Make the character an egg-head with knowledge of very long words to make NPCs do the dirty work.
Does the player run around without plan… well, you get the idea.
Example: The Elder Scrolls: Oblivion does this. How you talk to others, and how you overcome obstacles determines the character class you play. All in the game's tutorial.
XP
Not one of us suddenly gets insight into a problem area, and gains the knowledge how to solve more difficult problems than before. In fact, learning is the most tedious and error-prone process we can subject ourselves to. Ergo, it's boring. However, nothing says a game can't take over the leveling and skilling aspect for the player. The more a player uses certain skills, the better the character should become at doing the player's bidding. The more hits a character suffers, the more hitpoints a character should gain. Without player input.
Example: The Elder Scrolls Series comes closest. The more a certain skill is used, the higher the corresponding attribute can become.
(source: http://flickr.com/photos/54357394@N00/367219521 Note: Not an actual Elder Scrolls Game)
Karma
As everyone who grabbed a book on ethics probably knows, the concept of good vs evil isn't at all clear cut. A one dimensional system is woefully underwhelming to measure if a deed is "good" or "bad". A two dimensional system is a bit better. It allows to record the motivation for a deed, as well as the pure "good-/badness" of a deed. And ethics change from group to group, too! Cannibals have no problem with consuming human flesh, but the Order of Light and Hope will see that differently. But Cannibals aren't necessary evil, either. After all, they can oppose the Dark Lord as much as the Order of Light and Hope, while the Order's priests are corrupt to the bone.
Representing all this in a game is, in my opinion, overkill. A two dimensional system (good vs evil deed, pure vs. corrupt motivation) would be a large improvement already, and Good Enough.
Example: Fable II uses a two-dimensional ethics system, just as I described. What a coincidence, is it not?
TL;DR
A little fresh wind beyond the tried and true can't hurt, and my ideas aren't new, either. In one form or another, they've already been implemented in a game! Now, all that needs to be done is pulling all this into one game. I'd be intrigued by a game with these mechanics. It can only aid in immersion if we aren't pulled out of the game into a character screen.
I don't really like Karma or ethics systems in games at all... who cares if you're good or evil? Even if you do mostly evil things, when you go to a new town, how would they know? Just give the options and don't bother with the whole karma thing.
I do like the way Borderlands has simplified many of the RPG elements... you rank up and it lets you know when you have a skill point to apply - it is used on a very simple tree. The weapons are all still different and there is the usual RPG aspect of comparing stats on weapons and choosing the best - but again, they've kept it very simple.
I like the new "blurring of genres" and seeing RPG elements in other games!
As for character, it could be a good idea to suggest a class for you based on how you do the tutorial, but it should be that, a suggestion, not a forced choice. While in Oblivion they do offer you a suggestion, you're still free to choose your own class. On the other hand, how would this work out for games without a tutorial dungeon? I know I found it annoying to play through Oblivion's tutorial dungeon anytime I started a new character till I saved near the sewer exit.
As for XP, I don't think the current system is that broken, even if classic. What about the RPGs that net you XP like normal, but allow you to choose what skills and stuff to raise on level up? Oblivion sort of does that, although the "improve by use" system would have to be well designed. I felt Oblivion didn't always do a good job of this, like having your weapon skills increase by number of hits, not how powerful the hit was, so using a dagger would increase your Blade skill a lot quicker than using a claymore. It also led to things like casting the same weak spells over and over while traveling to improve them. Improving your skills could lead to some boring grinding. Sure killing monsters can be grinding in itself, but at least you're not doing actions that might be even more boring.
As for karma, oh boy, in order to do that justice would take serious work. I felt karma systems haven't been done very well, especially since they tend to favor good or evil over the middle ground, not to mention you get stuck with black or white choices pretty often. For one, a karma system would have to very from location to location, from person to person. That alone can get so complex that developers don't even bother and just go with a global karma system.
Basically, even your suggestions won't always change the old RPG formula, at least if not done correctly. There can be flaws in each of those new systems, along with the older ones.
Elsa, the use of Karma depends on the story to be told, IMO. It makes no sense in an Action RPG, ala Borderlands or Diablo. It does make sense in something that wants to invoke a sense of immersion, however, since it allows NPCs to react to your actions. However, it should be meaningful, and not a bullet point. By no means do I want Karma in everything.
A reputation system like in WoW is a nice choice, too. And not bothering with it makes sense, as well.
RBinator, very valid points. But I probably made mistakes in my examples, in that I wasn't clear enough. :)
So, to elaborate:
The game's "learning" of what you do doesn't necessarily have to be in a tutorial level per se. It can be the first 10 quests (with different options to approach them), or something like Jagged Alliance 2's "psychology test".
And of course you should be able to change it, if you so desire! I was more thinking of it as a suggestion that helps the new and/or immersive player (Fallout 3's pop up if everything was ok during childhood is a bit annoying).
As for XP: Well, in general I prefer a "spend XP on skills" approach, and dislike the "level up" approach. The trouble is, that the former is very, very difficult to balance, as I know from my pen & paper RPG experience. In an ideal game, I wouldn't even have to spend XP, since the game would "know" what I wanted to do. We are very far from that, tho. :P
And regarding Karma: Yup, absolutely, 100% agreement there.
Also: I'm not suggesting a silver bullet, since there is no such thing. I like my Action RPGs, and I like my D&D off shoots, and I like my Fallout 3s, and I recently replayed Fable 2, despite it being rather.. boring in a way.
I guess I'd like another, new, sort of RPG, that compliments the current servings, not to replace them.