Virtual reality has been the technology of the future for going on 30 years, and after a few stops and starts, we seem to be at the cusp of some really amazing immersive experiences. In fact, very good VR is already here if you’re willing to don a headset. Perhaps this type of paradigm shift will require us asking ourselves if we really do want to be fully transported to another dimension or galaxy, while blocking ourselves off to nearly any sort of external stimulus. Or maybe it’s a matter of not having one more piece of gear.
Besides having to actually put on a headset, VR experiences generally require some form of handheld controller or controllers. These track both your hand positions and act as a trigger for whatever action you need to perform. This works well if you’re playing a shooting game, or perhaps performing sim-gardening with a trowel or two. However, the reality of our human experience is that we don’t always have a tool in-hand. With ten fingers, there’s a lot more going on than can be described with one or two blunt instruments.
Controllerless VR with Leap Motion
Ideally, what you need for a truly immersive VR experience is a device that tracks your hands and fingers without any sort of controller, or even gloves. You’d also want it to be small enough to unobtrusively fit on a desk, or even be attached to a headset. While those goals seem loftier than the promise of VR presented in 1992’s Lawnmower Man, all of this is actually a reality, and has been for several years, in the form of Ultraleap’s Leap Motion Controller.
Such a device, however, seems almost too good to be true. Can a sensing setup shaped like a large thumb drive, at a cost of less than $100, actually deliver ten-finger, hand, and arm tracking? Let’s find out:
Leap motion tracks open and closed hands
Leap Motion Controller Setup
The Leap Motion comes in a rather unassuming box, with the unit itself, along with two USB-A to Micro USB 3.0 cables. Hardware installation is as simple as plugging it in, but its documentation notes that it doesn’t always work well with a hub, per power requirements (more on this later). Anecdotally, I found that the shorter cable, with presumably less resistance, seemed to work better. However, that could be a coincidence. It also has issues with bright lighting, so make sure to take that into account during setup.
Leap Motion Software
Calibrate!
Navigate to leapmotion.com/setup, as noted on the device’s peel-off covering, where you’ll be greeted with a desktop and VR option. There’s a V4 version of the software, as well as a V2 version for legacy desktop apps. I first chose V4, and after a calibration, was able to get it working with a demo program that tracks your hands and fingers. As seen in the first two images in this writeup, it almost appears as if it’s using some sort of augmented reality X-ray device.
Interestingly, it can track each finger (if not always 100% accurately) pretty well, even when some are partially occluded. The trick here is that it uses two cameras and infrared lighting to track images in stereo to infer three-dimensional input. Leap Motion puts it this way in a blog post:
"Despite popular misconceptions, the Leap Motion Controller doesn’t generate a depth map – instead it applies advanced algorithms to the raw sensor data… Our tracking algorithms interpret the 3D data and infer the positions of occluded objects. Filtering techniques are applied to ensure smooth temporal coherence of the data. The Leap Motion Service then feeds the results – expressed as a series of frames, or snapshots, containing all of the tracking data – into a transport protocol."
So, it’s not magic, or harmful radiation, but an advanced tracking algorithm. Through this data passed along to your computer or headset, various games and functionality can be controlled.
V2 Software Compatibility
TAKT-RHYTHM
After having some issues under V4, I decided to try out V2 for its compatibility features. This was installed, and I applied the hotfix found on their website. With that (and some fiddling with the plugs on my Windows 10 computer) I got it to work with the Leap Motion App Home with the Kyoto app, as well as the “TAKT-RHYTHM” game. This and other games can be downloaded from Leap Motion's gallery.
The game really highlights the Leap Motion’s abilities, as it invites you to hit and manipulate notes as they progress toward you through a square corridor. It’s similar to Guitar Hero, or perhaps Beat Saber sans controller, and is pretty fun until your arms get tired. The experience would certainly be better on a VR headset, and with a processor designed for 3D gaming, rather than my laptop.
There are also various hands-free computer control applications available for the system. Given today’s focus on keeping surfaces clean, such functionality could be extremely useful. You can even find some mention of using Leap Motion with an Arduino via Cylon.js, which presents a wide array of other possibilities.
We (Don’t) Have the Power!
After trying this out, I really think much of the issue with my setup was that my USB hub didn’t have enough power to spare, and my laptop’s single USB-A port wasn’t really up to the job either. Given the advanced processing going on, and the fact that it physically gets warm to the touch, I’d suggest making sure you have an independently powered hub if you have any issues with the device. Embarrassingly, it also wasn’t immediately clear how to control things with my hands, so make sure to put them over the controller, stretch them forward, and experiment before falling back to older software (or even giving up).
Leap Motion and Unity
Of course, the real magic of this type of system is making your own virtual worlds. Leap Motion provides assets for the Unity engine, as well as an official plugin for the Unreal Engine 4, meaning that you can create your own world, step inside, and poke around. It’s a rather amazing concept, and if you think it’s a bit beyond your abilities, check out the excellent Leap Motion and Unity Tutorial by Sean Horgan. In it, he’s able to create an interactive ball environment in roughly 20 minutes.
Sure, rolling a ball around is not the most interesting thing you can do on the computer, but it’s absolutely incredible that you can set up something like this is such a short time. Give yourself a few days with Unity and Leap Motion and the sky is virtually the limit of what you can do! It seems there’s no other hardware quite like the Leap Motion device, so if you’ve been interested in creating virtual controller-free interactions, there’s no better time to get started than now.
