Our janky eyes make all of it really hard
Back in the day, everyone thought virtual reality was going to be the future of games. We made movies, TV shows, and even very bad early attempts to make head-mounted displays. All of those devices failed, and over time, VR dropped out of our collective imaginations.
With the Oculus Rift just a little ways off, and more and more people in the gaming community getting their hands on demo units, Valve tried to port Team Fortress 2 to the thing and share what they learned from that experience to other developers in a pair of talks.
The human eye, it seems is very, very poorly suited to the new equipment. Heads-up displays, for example -- a mainstay of modern gaming titles -- are very difficult to implement on a screen held one or two inches from the eye. Peripheral vision is poor at that range, and the center of the eye, or the fovea must move cross almost three quarters of the average person's entire field of vision. For some people, that movement isn't even possible.
Valve representative Joe Ludwig speculated that, in the future, HUDs will be removed from dedicated VR games. It's too difficult and too distracting for people to implement them in classically designed titles.
At the same time, head rotation and eye movement marginally independently of one another, causing severe "VR motion sickness." Joe described this as being more similar to sea sickness than anything you might get from a car or plane. Over time, for example people develop "VR legs" and can adapt either significantly diminishing symptoms or eliminating them entirely.
Having seen this and tried the Oculus Rift, I'm amazed at how relatively not-sick I got. I'm usually pretty suceptible to motion sickness, but in playing the thing I only felt a little queasy. In time, and with folks like Valve trying to make adaptive software for these systems, it is completely concievable that we will have these problems licked. Then again, reps from the company stated that they didn't expect to have a complete solution inside this decade. So don't wait up for the VR revolution.