Back in March I submitted this content as part of a report and presentation on motion tracking systems in professional and consumer environments. As the internet is mostly full of hypothetical discussions of the consumer systems written before they were released, I figure I should put this up somewhere so that it might one day be informative to somebody (who will probably just copy it verbatim anyhow). The full body of work included observations on motion tracking in motion picture production by a collaborator, and comparisons of the professional and consumer motion tracking technologies.
The most recently concluded generation of home game consoles was notable for the onset of motion controls, such that by the end of it each of the three main competitors had one for their system. Each had a distinct way of operating, but faced similar challenge in balancing quality with cost to manufacture.
Nintendo Wii’s Wiimote
Of the seventh generation of home video gaming consoles, the Nintendo Wii launched first. Unlike its competitors, it focus on motion control from the outset and targeted a broad and mainstream audience.
The Accelerometer axis and motions of the Wiimote (Source: Osculator)
The key technology of the console was in the controller, the "Wii-mote". Within it was a chip (ADXL330) (Analog Devices
) containing an accelerometer system built around a 'Microelectromechanical' system (MEMS) that enabled measurement of the acceleration of the device in three axes. This information could be processed by the Wii for the movement and orientation (due to Earth's own gravitational field) of the Wii-mote in 3D space to be used in the software. On the end of the controller, which would be held away from the operator, was a simple monochrome camera optimised for detection of infrared wavelengths. In the same module on the PCB is built-in image processing technology for tracking up to four points that are most likely to be those of the sensor bar. When the two points are detected, the distance and orientation of the controller relative to the bar could be calculated. Assuming the bar was above or below the screen (and the console is told which) this allowed for calculation of the area on the screen at which the controller is pointed (with proper user calibration).
This made the controller a simple but effective tool for navigating guided user interfaces and for performing the actions required of the player for the multitude of games on the platform. Further enhancement to the motion control capability of the Wii included a secondary handheld attachment dubbed the "Nunchuk" which itself included an accelerometer chip (LIS3L02AL) which allowed interface with games that wanted to track the motion of both hands, with the motion and orientation relative to the Earth of the Nunchuk also made available to the software. Later on in the life of the console, at attachment for the bottom end of the Wii-mote was made available called the 'Wii MotionPlus". Inside this was contained the IDG-600 multi-axis MEMS rate gyroscope (InvenSense
). This was a more precise technology for tracking orientation than the accelerometers, which only inferred linear motion from analysis of the information, and in conjunction with them could produce better overall tracking of the Wii-mote.
At any given point in time the Wii-mote and Nunchuck accelerometers portray a snapshot vector of the overall force acting on due to a combination of gravity and any movement by the user, when processed with the previous vectors and points in time where the Wii-mote was able to detect the horizontal plane from the sensor bar, these snapshots can be used meaningfully by the software as instructions for avatars or GUIs.
The Signal Path for Wiimote Motion Tracking (click here for original)
Users of the Wii occasionally developed a form of Tennis elbow from extended play with the Wiimote, and the often violent actions necessitated the use of a wrist strap to prevent the controller being flung around by accident. The product was limited at the time by the desire to minimise production costs for the console to maintain a competitive edge, indeed the console was already light on graphic processing power to achieve this. As technology improves and gets smaller, better and smaller accelerometers and gyroscopes will be produced and implemented in more ways. Most modern smartphones will feature one, the other or both for sensing device orientation for screen reorientation, for controlling games available on the platform or even just allowing an extra control option. Although the Wii's motion technology was relatively low budget and insensitive at the time, it will continue to be relevant through the eighth console generation with the support of its peripherals and software library by the successor platform, the Wii U.
Sony released their Playstation Move interface 4 years after the Nintendo Wii and the PlayStation 3 itself were made available. Although they included a motion control system in their Dualshock 3 controllers for the platform, it was rudimentary and unpopular, few games implemented it and fewer gamers opted to use it. Sony opted for a highly similar controller and accompaniment to the Wii-mote and Nunchuck, with important differences that often took advantage of the higher processing power of the platform.
Notable differences between the Wiimote/Nunchuk and Move controllers is the lack of connection between the Move controllers, they can use separate Bluetooth connections due to the PlayStation 3 supporting up to seven paired bluetooth devices (compared to the Wii's four). The Move controllers do have mini-USB ports for charging like the Dualshock 3, and can be tethered to the console for charging during play. There are also more buttons across the Move controllers due to the need to replicate as much of the Dualshock 3 functionality as possible. The Move's additional 'navi-controller' does not track motion.
Perhaps the most immediately notable thing about the controller is the orb. This RGB LED illuminated bubble performs a similar role to the Wii sensor bar, in that it presents a distinct visual identifier for the camera part of the setup to identify and track.The colour it glows can determined by the whole picture seen by the camera, from which the most distinguishable colour is derived. The colour glown can also be used to differentiate between players or provide another form of feedback. The LEDs are capable of 24 bit colour resolution. The camera in this scenario also performing a not too dissimilar role to that of the simple IR camera on the Wii-mote. It tracks the location of the glowing ball. Where the Wiimote could only use the IR sources when pointed at the screen and would calculate distance from the spacing of the sources, the camera can track the orb as long as it is in sight and can determine its exact location in space according to how large it is in the captured image.
The camera is important in understanding the development of Move, because it precedes it by three years. The PlayStation Eye was released in 2007 as a successor to the EyeToy on the PlayStation 2, which is where developers first tested 'colour-based 3D wand tracking'. The success of the Wii prompted the revival of development and Move was developed to work with the Eye peripheral. Aside from Move functionality, it is also capable of basic face tracking.
Inside the Move controller is a multitude of microchips. A three-axis accelerometer (KXSC4) (Kionix
), a two-axis (X&Y) gyroscope (identified as STM LPR425AL), a single-axis (Z) gyroscope (Y5250H) and a three-axis magnetic compass (AKM AK8974). There is also a thermometer to enable adjustment calculations for temperature bias in the measurements. A microcontroller chip (STM32F103VBT6) and Bluetooth transmitter chip (Cambridge Silicon Radio BC4RE A16U) process and communicate measurements to the console, and receive back instruction for the LEDs. (Perl, 2012
) (PlayStation Move Teardown
The Signal Path for Move Motion Tracking (click here for original)
As with the Nintendo Wii-mote before it, the Move controller had to be designed to minimise cost. It does however show how much technology improved and cheapened over the four years since the Wii-mote, with more advanced motion-detecting chips being included. An obvious avenue for improvement would be increasing the data bandwidth of the camera, which will likely follow with future hardware. The PlayStation 4 was announced to feature the DualShock 4 with in-built Move functionality, achieved by a LED lit triangle facing away from the player (which will work like the orb on the Move controller) and presumably with similar or improved motion detection chips inside. The Move peripherals will continue to be supported by the PlayStation 4 too. A more advanced camera for the system, using a proprietary connection for higher bandwidth and compatibility, has also been detailed. (Sony
Microsoft’s Kinect for Xbox 360
Microsoft revealed the Kinect in 2009. In the years leading up to this they had licensed the camera array and a microchip from PrimeSense, and acquired 3DV systems and Canesta to acquire patents associated with them. It launched in 2010, barely a month after the PlayStation Move, with $500 million poured into the marketing - more than the console's own launch received! It paid off as Kinect claimed the record for fasting selling consumer electronics device.
Kinect (an amalgamation of kinetic and connect) was distinguished from its competitors by featuring controller-free gaming. It is just a sensor.
The Kinect features three windows, though only the centre two are cameras. The off-centre one is an infrared laser projector which blankets the area of play in over three hundred thousand dots or specks. This array is very precisely designed and programmed into the Kinect for recognition by the centre-right camera, which is sensitive to Infrared wavelengths. The specks of light will have a different focus depending on how far from the sensor they are, ie the further they are, the blurrier they are. This is a method of depth perception called 'depth by focus'. The Kinect also employs a method of depth perception called 'depth by stereo' by having the sensor which is imaging the pattern a known distance away from it, so that the deformation of the pattern on the environment can also be used to infer distance and shapes. The middle camera is a simple RGB one and used for imaging the environment in the visible spectrum. (Reichinger, 2011
The depth image data is used to identify shapes of players, using basic awareness of what shapes people come in and the proportions they usually conform to. The Kinect effectively learns about a player the more they use the sensor, as their biometric data is stored and associated with them for future reference. RGB captured data such as their facial features are also stored for functions such as signing them into their system user accounts.
Inside the device, there is a proprietary chip from PrimeSense (PS1090-A2) which processes the depth or RGB image (whichever is needed), an accelerometer (KSXD9) to detect orientation for calibration and also detect if the peripheral has been moved (which would necessitate recalibration) and a motorised sensor tilt for automated calibration. (Microsoft Kinect Teardown
The Signal Path for Kinect Motion Tracking (click here for original)
The original prototype for Kinect cost Microsoft around $20,000 to produce, and so to develop that into a device that could retail for less than a hundredth of that compromises had to be made. The RGB and depth cameras are the lowest resolution possible for an acceptable result, this reduction in data bandwidth greatly reduces processing power required but also impacts on the level of detail resolvable in the skeletal tracking. Kinect cannot track individual finger movement. The field of play in Kinect is also restrictive both in the minimum space needed and the limited number of people who can fit into the space of play. It is expected that Microsoft will improve on this with their next-generation gaming console and peripheral. Codenamed 'Durango, it is rumoured to track more players (larger field of play), more skeletal joints per player (better image resolution and processing) and better RGB image quality.
Comparisons and Conclusions
Motion control emerged in the seventh generation of console gaming, and apparent quality and power of the systems did not reflect success. The eight console generation will be fought heavily with motion controlling as the graphical quality possible plateaus and other ways of differentiating each console from competitors are required.
Though the Nintendo Wii kick started the Motion Control race, I feel that Microsoft and Sony both outshone them. PlayStation Move had vastly superior motion tracking to the Wii, and Kinect brought full body motion tracking to the consumer market for the first time despite the corners cut to do so.
Xbox One has been announced and detailed along with the sometime controversial Kinect attachment. As with the majority of information about the original Kinect, most currently available information is conjecture and marketing. It has a higher resolution camera and better skeletal tracking of more individuals as was assumed. It also, to my interest, uses a different method for inferring distance to the original sensor. Instead of depth by focus and depth by stereo, it uses a method known as 'time-of-flight' where the time taken for the infrared beams to hit the environment and return to the sensor is used with the known speed of light in the medium to calculate distance. This requires a much more powerful sensor to be executed accurately, making the sensor somewhat impressive in that aspect. Not using depth-by-stereo is why Kinect for Xbox One only has one 'eye'.
It has also become apparent that the sensor for PlayStation 4 Move uses two cameras. This will improve tracking greatly, and may have been a necessary improvement to enable accurate tracking of the DualShock 4 light bar.