Every camera makes a compromise. Shoot in near-total darkness, and there may simply not be enough photons to form a clean image. Add film grain for texture and atmosphere, and you've made the footage harder to compress. Work with lower-resolution source material, and you're fighting against the expectations of today's displays before a single frame is delivered.

These look like separate problems. But they share one fundamental question: how much useful visual information can we recover when what was captured is incomplete, noisy, or difficult to process? Across the imaging pipeline — from the moment light reaches a sensor to how video is ultimately viewed — Dolby researchers are working to recover what cameras leave behind.

The goal lies in recovering the right detail: motion that stays consistent, creative intent that stays intact, and perceptual quality that holds up on the screens people watch.

Seeing in Almost Total Darkness

Imagine a camera that counts individual photons. Although single-photon detectors already exist, operating at extraordinary frame rates and capturing information in conditions is where conventional cameras simply give up. When very few photons are available, the data they produce is extremely noisy. The signal is real, but it's buried.

Quanta Diffusion — QuDi — was built to solve that problem. Rather than asking a generative AI model to freely invent missing detail, QuDi inserts a physics-based probabilistic sensor model directly into the reconstruction process. It understands, mathematically, how a single-photon sensor captures light. A learnable optical-flow component runs alongside it, tracking motion across frames in real time.

That combination breaks a trade-off that had long constrained low-light reconstruction: use more frames to reduce noise, and motion becomes harder to align; use fewer frames, and the noise becomes harder to remove. QuDi does both simultaneously.

Trained on simulated 3-bit quanta frames at conditions around 1 lux — roughly the light level of a candle across a dark room — the system delivered an average 2.4 dB PSNR improvement over leading approaches across 31 test videos. Tests on real sensor data captured at 10,000 frames per second showed even stronger results during fast motion.

However, at the extreme low end of the light spectrum, temporal inconsistencies remain. But the direction is clear: the physics of capture and the intelligence of reconstruction are more powerful together than either.

Removing Grain Without Erasing the Filmmaker's Vision

Here comes the next challenge. Film grain started as a physical artifact of analog film, but it became something more — a texture, a mood, a deliberate creative choice.

It also makes video harder to compress. Grain is random, high-frequency, and largely unpredictable — exactly the kind of signal that video codecs struggle with. Removing it before encoding can meaningfully reduce the number of bits required, but the result is a footage that looks like it's been scrubbed of its character.

Dolby's approach starts with a different question. Rather than immediately removing grain, the system first asks: how much grain is here, and where?

A Noise Strength Estimation Network analyzes five neighboring frames and produces a spatially varying grain map — a precise picture of where grain is heavy, where it's light, and where it shouldn't be touched. That map then guides a second network that removes grain selectively, with self-attention mechanisms preserving the spatial and temporal details that matter.

Compared with established video-denoising approaches, the method delivered improvements of approximately 4 dB PSNR while maintaining temporal stability across frames.

This restoration technique works better when a system understands what it is removing, rather than treating every high-frequency signal as generic noise. Grain that was added with intention also deserves to be removed with equal care.

The Bigger Challenge: Recovering Resolution

Can AI reconstruct high-resolution videos from lower-resolution input? A Dolby-led review of 60 neural-network-based approaches and eight traditional techniques examined the question from every angle: architectures, motion-compensation strategies, visual quality, inference speed, model complexity.

The honest answer is that it's hard. Recovering missing detail is an ill-posed problem — multiple high-resolution images could plausibly correspond to the same low-resolution input. There is no single correct answer, only better and worse ones.

But the stakes make it worth pursuing. Video represented roughly 70% of mobile data traffic in 2022 and is projected to approach 80% by 2028 — around 324 exabytes of data moving across networks every month. Better reconstruction doesn't just improve how footage looks. It changes how efficiently video can be distributed at scale.

Recovering What the Camera Couldn't Capture: How AI Is Rebuilding Better Video

{{ pageEducationalSlot.headingContent[index] }}