Car dashboards have become digital cockpits full of buttons and displays for a whole of infotainment systems, but they’re on the road to be touchless thanks to gesture recognition technology.
Speech recognition already provides a human-machine interface that makes it easy to request information or perform a task while driving, such as asking a GPS device to plot a route to a desired destination — but it does have limitations. Both speech and gesture recognition fall under the category of a touchless user interface (TUI). As much as drivers are used to hitting buttons and becoming more accustomed to voice-activated features, gesture technology offers many advantages, including improved safety.
What’s Driving Gesture Recognition Technology in the Car?
There’s no shortage of driver distractions inside the modern vehicle and on the road, which is why touchless gesture recognition is increasingly being seen as a viable alternative to improve safety and comfort.
Aside from the rising use of smartphones while driving, adjusting the various environment and infotainment functions of the modern vehicle creates many opportunities for distraction — simply changing the radio station can take the driver’s eye off the road in a second, and a lot can happen on the road during the blink of an eye.
Voice recognition technology is an obvious solution because it allows drivers to keep their eyes on the road while dictating commands. However, it does have limitations; in part because it’s powered by artificial intelligence (AI). Depending on the training, the number of commands may be restricted.
Some speech-recognition technology struggles to recognize languages other than English, while others struggle with heavy accents. Different speech recognition engines have popped up to handle different languages, which leaves a somewhat fragmented market. Regardless of the native tongue, the quality of the microphone can affect how well speech-recognition technology inside the car performs, as does background noise.
Even as these challenges are addressed with improvements in technology, speech recognition doesn’t address the needs of deaf and mute drivers and passengers. In addition, having to repeat commands multiple times can become a distraction while on the road. Another benefit of TUIs is the reduction of wear and tear on the devices or components. In addition, a driver can make a gesture while wearing gloves, but struggle with a control panel that requires touch.
Just Gesture and Drive
Gesture recognition technology comes in many forms, but essentially works by recognizing human movement as the method of input. That input is detected by sensors and cameras that monitor people’s motions — in the case of automotive applications, it’s most likely the driver. Specific movements correspond with a command.
Aside from automotive applications, gesture recognition technology is being used for other human-machine interfaces, including smartphones and tablet PCs, industrial control panels, and video game consoles. Depending on the device and use case, it may be possible for users to set their own gestures. There are many basic functions in the automobile that could be done with gesture technology instead of by touch or voice, including environmental controls such as heat and air conditioning, music selection and volume, GPS navigation, and handling voice calls. Depending on how advanced the vehicle cockpit is, a driver could use a gesture to transfer applications from the main display to the instrument cluster and back again.
Hand gestures that control music/audio and incoming calls are one of the early examples of gesture recognition in an automobile, while a more advanced function would be the ability to use gestures with telematics systems, so the vehicle could provide information about nearby landmarks pointed out by a driver or passenger. Gesture recognition technology could even recognize when a driver is nodding off or in distress due to a sudden health issue, enabling a semi-autonomous vehicle to pull over safely and call for help.
Simple Motion Detection Requires Complex Systems
No matter the device or the environment, gesture recognition technology relies on sensors and cameras, algorithms, and AI to detect specific gestures and act accordingly based on system training. Foundational to a gesture recognition system is a camera with unobstructed views of a 3D area within the vehicle cockpit — likely on the roof. The area is illuminated by infrared LEDs or lasers, so there’s a clear image regardless of the available natural light.
Multi-modal sensors that combine RGBD data from the hand area with the movement data from the upper-body skeletal can improve classification accuracy. More reliable and precise detection can also be obtained with a single optical sensor that measures an object’s location, distance, and gestures — a single-sensor solution requires fewer components and reduces the complexity and cost for system designers.
Computer vision and machine-learning technologies powered by algorithms and AI analyze the gestures in real time and translate them into commands in real time based on a library of hand motions already on file. While there are variations demanding on the system designer and vehicle, common gestures include a pointing gesture to accept a call, while a call can be rejected with a swiping motion; spinning a finger can turn the radio volume up or down, depending on the direction.
Over the past decade, the sensors and algorithms have improved, making it possible to apply color and depth cameras in gesture recognition. However, there’s a lot of work to be done if gesture recognition technology is going to become ubiquitous in the modern vehicle. Drivers need to be comfortable with it, which means the gesture language needs to be standardized so it’s consistent regardless of vehicle and geographies.
As simple as a hand motion might be, implementing a gesture recognition system requires advanced machine-learning algorithms and intensive computing to process and recognize the hand motions that are captured by sensors and supporting hardware. Accuracy is critical, which means the systems for each domain require a great deal of data to train machine-learning models and compensate for environmental factors that might lead to inaccurate recognition, such as insufficient light.
Beyond the technology, there is a social acceptability element to gesture recognition technology: Drivers must be willing to perform the hand motions and sign languages. Like voice-recognition technology, gesture recognition technology will have its own adoption curve.
In the short term, the integration of gesture recognition technology into automotive cockpits will likely appear in higher-end luxury vehicles. Recent research estimates that around 40 percent of luxury car brands in Europe have implemented gesture recognition technologies for faster access and an improved and safer driving experience. But like other automotive advances such as rear-view cameras, which are now common on all vehicles, gesture recognition will become a standard feature over the longer term as the technology is perfected and becomes less costly to design and implement.
