Back in 2007, when Apple announced the first iPhone, Steve Jobs wanted to get rid of the stylus for touchscreen input. His argument was that a stylus is a horrible way for operating a phone. Because, it’s a separate, small, item that you need to carry around with you and it’s easy to lose. Instead they used the best controllers in the world: our own fingers. VR & AR use a variety of controls. From different type of controllers to gloves. However, one technology is revolutionizing how we operate our devices: hand tracking.
A lot of different devices support hand tracking. One of them is Leap Motion, a leading brand in hand tracking technology. One of the interesting features of Leap Motion is the possibility to mount the device directly to a XR system. This way the area in front of the device, for example the VR headset, is being tracked. However, Leap Motion isn’t the only company producing hand tracking devices. Some companies also produce gloves or exoskeletons.
What is Hand Tracking?
Sensors and receivers track the human body to measure its movement. The trackers recognize the gestures made by the users. In software this can be used to control menus, interact with your environment and navigate the world.
Hand tracking is great because it can eliminate the use of physical controllers. You just use your hand like everything is one big touchscreen. This is especially useful for inexperienced users. You don’t need to explain how the controller’s work.
When using VR, it is frustrating when the controllers are difficult to understand. Taking off the headset to look at your controllers destroys the experience. Using your hands, you don’t have to take off the headset because the way they work is very intuitive. Hereby more people can take advantage of VR technology. Grabbing a model and rotating it just like a physical object or stretching it to make it larger just like in real life. When using VR, you want to simulate normal human mimicry because it makes the experience realistic.
Exoskeletons
There are also different kind of gloves. These gloves use a reverse external skeleton to give feedback to the user. Gloves give the sensation of holding a physical object without holding it. A company called Sense Glove is making these advanced gloves for VR.
In the summer of 2018, we first got our hands on the device. Our CXO Robbin Geerlings got a demo and told us this. “What is great about this technology is the extreme realism. This is just a first edition but imagine it in 5 years from now. These will become smaller and more affordable. In the demo we experienced two different objects, an egg and a stress ball. Just like in real life the egg is fragile. Squeezing too hard will break it”.
This is just one demo application. In 5 years from now this technology will be even more advanced. We can’t wait!
The Mirrorworld
We briefly talked about the mirrorworld in our blog about the HoloLens 2. In short, the mirrorworld is a digital twin of the real world. Everything is digitized and can be searched for by machines. For example, projecting information in the real world instead of on your phone.
Hand tracking in combination with XR are making the mirrorworld possible. Imagine having a digital copy of your car. Using your hands, just like in the real world, you can learn about the engine. Grabbing a virtual tool and start working on the car. Controllers can’t do this in the way hand tracking can.
Surgeons in school can benefit greatly from XR and hand tracking applications. Practicing an operation in the virtual world is cheaper and can be used repeatedly. Research has shown that serious medical games are super effective. Hand tracking gives the surgeon more precise control of the game.
The Future of Hand Tracking
We can accurately track our hands and fingers. Leap Motion is an example of a good working hand tracker. Unfortunately, beyond a certain area the tracking stops. This generally is the area right in front of where the device has been mounted. This can be a problem because it limits what you can do.
Current hand tracking is accurate, but not in every possible situation. For example, if you hold your hand vertical the tracker will not detect all of your fingers. Because of this the experience may not work like you would expect it, or feel less realistic.
We believe this can be resolved by using light and heat detection. Your body sends out heat which can be used to increase the accuracy. Also, software will become smarter in detecting real objects. For example, if your hand hits a wall the software generates a sensation of impact. In this case you can feel objects just like in real life.
Some people may not be able to use physical controllers. Elderly people can benefit greatly from XR technology. But, explaining how a controller works can be a challenge. Hand tracking, when used correctly, doesn’t need an explanation.
Just like we talked about with eye tracking, hand tracking gathers data. Valuable data that can be used to analyse all sorts of things. Maybe tracking will become so accurate that we can detect Parkinson’s disease using algorithms in an early stage and help people more effectively. This would revolutionize the medical industry and help a great number of patients.