What good is speech that sounds clean but says the wrong thing?

For any great-sounding audio, audio creation and restoration are crucial. By combining semantic knowledge distillation with masked audio/acoustic modelling, we demonstrate how systems can generate and restore speech that is not just natural, but also intelligible and semantically faithful.

Our researchers highlight how speech AI needs to get the words right and not just the sound.

Teaching Machines to Understand Before They Speak

When you consider text-to-speech (TTS), most TTS systems traditionally follow two steps – identifying the meaning of the text and generating the audio. Dolby's approach collapses this into a single step, making the system simpler and faster, while narrowing the intelligibility gap that single-stage systems typically suffer.

The other challenge was speech restoration — repairing audio damaged by noise, reverb, or clipping. Building on Dolby's earlier MaskSR model, the new MaskSR2 adds semantic distillation to better preserve what was said during restoration. The result showed a measurably lower word error rate (WER) at the same model size and inference speed.

Dolby deploys the same core technique to address both challenges: masked audio modelling combined with semantic knowledge distillation. The speech AI model learns to "fill-in-the-blanks" of existing audio where the model learns to predict hidden parts of speech, while a smarter pre-trained "teacher" model keeps it anchored to the actual meaning of what's being said.

Through both techniques, our researchers observed improved intelligibility over their baselines. The TTS model outperformed single-stage competitors on speech quality, naturalness, and speaker similarity. MaskSR2 reduced WER while maintaining strong perceptual quality. The results point toward a clear direction for speech AI: separating semantics from acoustics using masked prediction as the learning engine for both.

As models get more efficient and semantic anchoring matures, this approach could underpin the next generation of voice interfaces, hearing aids, conferencing tools, and beyond.

Smarter Speech AI: How Dolby Is Teaching Machines to Say the Right Thing, the Right Way

{{ pageEducationalSlot.headingContent[index] }}