Carnegie Mellon Researchers Use the Body’s Acoustic Signatures for Gesture Sensing

Researchers from Carnegie Mellon have developed a platform that takes bio-acoustic interference patterns from inside the human body and translates them into expressive gestures that could be used for human/computer interface applications. Accurate gesture sensing is a challenging task to accomplish, and most systems (Kinect, Oculus, Vive, etc.) rely on external cameras and hand controllers to gain a relative position of a user’s hands and body. While those platforms are excellent in their own right, they can’t garner detailed information on what your hands and body are actually doing.

Carnegie’s new on-body gesture sensing technique is known as Interferi, which uses acoustic interferometry waves to extract data on what the body is doing. The research team used ultrasonic transducers positioned on the skin to create acoustic interference patterns from inside the body — in this case, the hands and face of the wearer.

For the Interferi platform, the team designed a pair of wearable devices, each with their own sensor configurations, which they describe “were used to identify useful transducer arrangements and machine learning features.” The hand and face prototypes were capable of supporting eleven and nine-class gesture sets (or poses) at 93.4 % and 89% accuracy respectively. Moreover, they were able to test their platform with four continuous tracking tasks, including smile intensity and weight estimation while holding weighted objects, and found the system never exceeded a 9.5% error rate.

Although the Interferi platform would be an ideal solution for HMI/robotics applications, it could also be used in others, such as gaming and VR, where discrete gestures would be beneficial for remote medical procedures. That said, the system needs per-user calibration before it can be utilized, and isn’t yet robust enough for applications beyond lab use, meaning we probably won’t see Interferi on store shelves any time soon.