Dhruv Jain earns ACM SIGCHI dissertation award for research on new systems for sound awareness

Dhruv Jain, assistant professor in Computer Science and Engineering, received the Outstanding Dissertation Award from the Association of  Computing Machinery’s Special Interest Group on Computer–Human Interaction (ACM SIGCHI). The award recognizes the most outstanding research contributions from recently graduated PhD students within the HCI community. Jain’s dissertation, “Sound Sensing and Feedback Techniques for Deaf and Hard of Hearing People,” presented novel sound awareness systems for deaf or hard-of-hearing (DHH) people.

Sound awareness is of critical importance for DHH people, encompassing safety critical sounds like fire alarms and sirens and  more mundane but useful cues like microwave beeps or door knocks. Currently, solutions to translate these sound cues into a form they can use have been limited in scope.

“The current solutions used by DHH people—such as flashing doorbells or vibratory alarm clocks—only substitute for some specific sounds, but do not provide a general awareness about environmental sounds,” Jain writes.

Jain set out to understand the broader sound awareness needs and preferences of DHH people, designing systems to enhance sound and speech awareness in multiple contexts. His research made use of iterative design techniques involving lab and field studies with DHH study participants.

In his first area, Jain explored how to classify and visualize sounds around the home. He created HomeSound, an Internet-of-Things based system to visualize sound activity in the whole house. HomeSound makes use of a network of connected “picture-framed” displays that run a deep learning engine and can be placed in different rooms of the house. Each display senses and classifies 19 common sounds in the home, like a barking dog or doorbell, and interacts with a centralized server to produce a single across-home visualization of in-home sound activity.

Jain further explored how sound awareness could be provided accurately by mobile devices, in particular using smartwatches. His team developed SoundWatch, a smartwatch app that detects and provides text descriptions of nearby sounds.

One of the key features desired by DHH people surveyed by Jain was more customizable or personalized sound recognition. This would allow end users to adapt a system to the most common sounds they interact with in their day to day life. To that end, Jain developed ProtoSound, an interactive system that allows users to personalize a sound recognition engine by recording only a few training samples (e.g., five for each sound). The key idea behind the system is that by training and evaluating a model repeatedly on datasets of limited training samples, it learns to train rapidly from a few samples in the field. Jain’s evaluations demonstrated that the model could accurately learn sounds in homes, restaurants, grocery stores, urban streets, and parks.

Finally, Jain worked to improve speech awareness with augmented reality (AR). He designed HoloSound, an AR system that provides real-time captioning, sound identity and sound location information to DHH users via a wearable device. The system makes use of a HoloLens and external microphone array, providing directional sound awareness, a short list of recent environmental sounds, and transcription of speech that can be customized and repositioned in the field of view.