A person walks across the screen and their movement is slightly off — not wrong enough to name, but wrong enough to feel. A door slams, and the sound arrives a beat too late. An object falls and ignores gravity in a way that no real object ever would.

These aren't technical failures in the traditional sense. There are no compression artifacts. No obvious glitches. The image is sharp. The audio is clear. And yet something in your brain quietly registers: this isn't real. That moment — the instant the illusion breaks — is exactly what Dolby researchers are working to understand and eliminate.

What Makes Generative Media Feel Real?

Generative AI has crossed a remarkable threshold. It can produce video from a line of text, conjure imagery from a handful of frames, and synthesize sound from silent footage. The raw capability is no longer in question.

But capability and believability are different things entirely. When sight and sound combine, the whole experience needs to feel as though it was made together, not assembled from separate parts.

Dolby researchers observe this challenge from two directions:

One investigates how people perceive realism in generated audio-visual content — specifically, where and why the illusion fails.

The other explores how AI can generate sound that doesn't just match what happens on screen, but when it happens.

Together, they point toward a future where generative media is judged not by what a model can technically produce, but by how convincingly the pieces hold together.

Dolby researchers studied these failures through human perception — because ultimately, it's humans who decide whether something feels real.

In a controlled study, 36 participants compared pairs of five-second clips and selected which appeared more realistic. The experiment paired 66 real user-generated videos with 66 AI-generated equivalents. Eye tracking revealed not just whether viewers noticed something was wrong, but where they looked when they did.

When participants compared real video and audio against fully AI-generated content, they correctly identified the real version 93.4% of the time. Remove the audio entirely, and accuracy barely moved — still around 93%. But when the imagery was identical and participants had to distinguish real microphone audio from AI-generated audio, accuracy dropped to 82%.

Understanding where realism breaks is one challenge. Building systems that don't break it is another.

Consider a simple scene: someone strikes a tennis ball. An AI system might correctly generate the sound of a racket making contact. But if that sound arrives even a fraction of a second before or after the visual impact, the effect collapses.

This is Synchronization — the precise alignment of sound to visual event — is one of the most demanding problems in generative audio. And it's the problem that MaskVAT was designed to solve. Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity  treats audio quality, semantic relevance, and synchronization not as separate goals, but as connected ones.

The model trained on approximately 155,000 ten-second videos and was evaluated across nearly 15,000 test clips — including footage of live instrument performances, where the synchronization demands are unforgiving. Every visible note, every struck string, every percussive hit needs a sound that arrives at exactly the right instant.

MaskVAT was evaluated across three dimensions simultaneously: audio quality, semantic relevance, and temporal alignment.

The hybrid architecture delivered the strongest overall balance, outperforming comparison systems in synchronization and overall quality while remaining competitive in fidelity and relevance. A separate listening study with 19 participants — including 11 audio-processing experts — reinforced the synchronization advantage.

One maps the territory of human perception — charting exactly where and why generative media loses its hold on the viewer. The other uses those insights to build models that close the gap, starting with one of the most perceptually damaging failures: sound that doesn't land on time. For Dolby, this is a natural extension of decades of research into how people experience sight and sound together — and what it takes to make that experience feel not just impressive, but true.

That creates a productive loop:

Understand human perception → identify what breaks realism → design models around those perceptual priorities → evaluate the complete experience.

Because the future of generative media won't be defined by whether AI can create convincing pictures or impressive audio in isolation. It will be defined by whether everything we see and hear works together well enough that we stop noticing the technology at all.

When Sight, Sound, and Timing Work Together

{{ pageEducationalSlot.headingContent[index] }}