The emotional and structural experience of music remains a significant accessibility challenge for the deaf and hard of hearing community. MUSTEM (Multisensorial Emotional Translation) is a novel system designed to translate music into a rich, coherent, and scientifically-grounded sensory experience. This project presents a dual-modality approach addressing this challenge through two interconnected components.
First, a low-cost, portable hardware prototype performs real-time audio analysis, mapping distinct frequency bands (sub-bass, bass, mid-range, treble) to a four-channel vibrotactile system, allowing users to feel music's rhythmic and foundational structure. Second, to overcome the processing limitations of embedded hardware, a high-fidelity software simulation demonstrates the full potential of the visual translation. This assistive dashboard decodes musical components—such as rhythm, harmony, and frequency spectrum—into an intuitive and educational visual interface.
MUSTEM offers a comprehensive framework for sensory substitution, presenting a viable and accessible pathway for the deaf community to experience music not just as vibration, but as a structured, substantiated and emotionally resonant visual and tactile language. Current technologies often focus solely on conveying rhythm through vibrotactile feedback, frequently overlooking the rich harmonic, melodic, and timbral content that constitutes the emotional core of music. MUSTEM addresses this gap by providing an integrated, low-cost framework that offers a holistic and informative translation grounded in scientific principles of psychoacoustics.
This project was presented at the International Conference on E-Health and Bioengineering (EHB 2025), where it received an Honorable Mention for its innovative contribution to assistive technology and accessibility. The work is currently in the process of being added to the Springer Nature proceedings.
Assistive Technology · Sensory Substitution · Music Visualization · Vibrotactile Feedback · Psychoacoustics · Human-Computer Interaction · Accessibility · Deaf Community