AI-generated video looks almost right — until you notice that in some cases, the movement doesn't quite match the music. The body moves to the tune, but it's not really dancing. This gap between motion and sound is harder to fix than it looks and Dolby have a solution exactly for that.

Our researchers built Rhythm Fusion — a model takes music as input and generates full dance skeleton sequences to generate movement that isn’t just plausible, but it genuinely tries to fit the tune beat by beat without it appearing strange.

The secret is two specialized components working alongside the main generator:

A Fusion Sync Classifier continuously checks whether the motion and audio are actually in sync.

A Fusion Sync Enhancer that goes deeper by learning the structural relationship between sound and movement. That way, the two stay connected not just frame by frame, but across the entire sequence.

Both the Classifier and Enhancer keep the generated motion anchored to the music in a way earlier models simply couldn't. This model was trained on the AIST++ dataset — over 1,400 sequences spanning 10 dance genres — giving it a broad foundation to work from.

When tested against leading models including FACT, Bailando, and EDGE, Rhythm Fusion came out ahead on every metric: more realistic foot-ground contact, closer motion distribution to real dance, and stronger beat alignment.

The result didn’t appear robotic. The movement responded to music the way a person would. Looking ahead, the team sees potential beyond dance entirely, extending this approach to any scenario where sound and motion need to move as one — from music visualization to creative AI applications.

Move to the Music: How Dolby Built an AI That Dances in Sync

{{ pageEducationalSlot.headingContent[index] }}