When you watch a guitarist's fingers fly across the fretboard, your brain connects every movement to every note instantly. However, getting AI to do the same isn’t easy.

Most audio-visual AI systems traditionally compress video and audio into a single summary—one flat snapshot of what happened before matching audio and visuals. It is efficient, but it often misses the fine timing and motion that make media meaningful: the tempo of a riff, the rhythm of a drumstick, the precise sync between sound and motion.

Dolby researchers revisit how machines learn from sound and vision together.

Dolby Taught AI to Truly Listen and Watch at Once

Their approach: SCAV (Sequential Contrastive Audio-Visual Learning) that keeps the full sequence intact. Instead of collapsing everything into one embedding, SCAV compares audio and video moment by moment—preserving the fine-grained temporal detail that standard methods discard. This preserves a richer context and enables the model to capture subtle cues across time.

Tested on two challenging datasets — VGGSound and a music performance dataset — SCAV delivered up to 3.5× better retrieval accuracy than traditional methods, using fewer parameters and less training data. To make it practical at scale, the team also built a hybrid retrieval mode that combines speed and precision, cutting compute time from 14 GPU seconds down to under one.

This is just a start. In hindsight, if multimodal AI is going to understand real media, it may need to treat sound and visuals less like labels to be matched and more like experiences that unfold together. That has clear relevance for retrieval today and for richer multimodal generation and understanding in the future.

Rethinking Audio-Visual Learning at Dolby: How Sound Finds Its Scene

{{ pageEducationalSlot.headingContent[index] }}