The Right Semantics at the Right Time:
Adaptive Inference for Semantic-Faithful Video Diffusion

Standard inference asks one conditioning regime and one timestep allocation to serve different generative responsibilities across the denoising trajectory. Our framework instead couples semantic granularity with where the sampling budget is spent, enabling coarse scene organization and fine-grained constraint realization to receive stage-appropriate support while preserving a stable generative trajectory.

Method Overview

Adaptive two-stage video diffusion inference with semantic-gap-adaptive prompt scheduling, two-stage sampling-budget reallocation, and coarse-to-fine transition guidance — The framework organizes a single denoising trajectory into two complementary semantic stages. A coarse textual proxy derived from the input prompt retains primary objects, spatial layout, global motion, and other coarse-grained semantics, while the original prompt remains the final semantic target. Semantic-Gap-Adaptive Prompt Scheduling determines when generation should move from coarse semantic establishment to fine-grained realization according to how much additional semantic content is present in the original prompt. The first stage uses denser high-noise updates to establish the main scene configuration. At the stage boundary, Coarse-to-Fine Transition Guidance uses the prediction difference between the two text conditions to guide the intermediate latent toward a higher-noise state that can accommodate the additional semantics without discarding the coarse configuration already formed. The second stage then uses denser low-noise updates under the original prompt to realize local events, appearance attributes, background details, and other fine-grained constraints.

ⅠAdaptive Two-Stage Inference Framework

input prompt p

→

coarse textual proxy c

From the original input prompt p, a language model derives a coarse textual proxy c that retains primary objects, spatial layout, global motion, and other coarse-grained semantics while omitting local events, appearance attributes, background details, and overly specific modifiers. The first stage pairs c with denser high-noise sampling, where the latent remains highly flexible for establishing the main scene configuration. The second stage pairs p with denser low-noise sampling, where updates act more directly on local interactions, appearance, and background details. In this way, the semantic responsibility of each stage is matched to the region of the denoising trajectory that provides the most suitable sampling support.

ⅡSemantic-Gap-Adaptive Prompt Scheduling

g = 1 − cos(E(c), E(p))

This module determines the switching point from the coarse textual proxy c to the original input prompt p for each input. Here, E(·) denotes the text embedding model, and E(c) and E(p) are the sentence-level embeddings of c and p, respectively. Their cosine similarity cos(E(c), E(p)) measures how closely the two text conditions are aligned, and the semantic gap g = 1 − cos(E(c), E(p)) is used as a relative proxy for the semantic increment from c to p. A larger g indicates a greater semantic increment and therefore moves the switch earlier, leaving more of the remaining denoising trajectory for the additional constraints in p; a smaller g keeps the coarse stage active for longer. We clip g to a predefined effective range and bound the mapped coarse-stage ratio to prevent extreme similarity estimates from producing an excessively early or late transition, which could make either denoising stage impractically short.

ⅢCoarse-to-Fine Transition Guidance

v_guided = v_c + λ(v_p − v_c)

At the switching boundary, the current latent has already followed a denoising trajectory shaped by the coarse textual proxy c, so directly replacing c with the original input prompt p can make the new conditional updates incompatible with the current state. Let v denote the model prediction that determines the text-conditioned latent update at a given latent state and noise level. Evaluating the same state under c and p gives v_c and v_p, respectively, and their difference v_p − v_c captures the change in the denoising update induced by the additional semantics in p. The guidance scale λ controls the strength of this conditional increment. Coarse-to-Fine Transition Guidance uses the resulting guided update to steer a controlled transition toward higher noise, where the latent regains flexibility for fine-grained realization.

ⅣSampling-Budget Reallocation for Two-Stage Inference

Stage 1 · denser high-noise updates

→

Stage 2 · denser low-noise updates

This module redistributes a fixed set of sampling timesteps so that each semantic stage receives denser updates in the noise region most relevant to its role. In the first stage, more sampling points are placed toward high noise, where the latent remains highly malleable and denoising under the coarse textual proxy can establish primary objects, spatial layout, and global motion. In the second stage, more sampling points are placed toward low noise, where denoising under the original prompt acts more directly on local events, appearance attributes, background details, and other fine-grained content. The reallocation preserves the number and order of timesteps, redistributing the same inference budget rather than increasing the number of model evaluations.

Two-stage sampling-budget reallocation with denser high-noise updates for coarse semantic establishment and denser low-noise updates for fine-grained constraint realization

Baseline vs. Ours

Within each backbone, single-stage input-prompt inference and adaptive two-stage inference are compared under the same prompt across six semantic criteria.

Ours

Ours

More Qualitative Results

Additional within-backbone comparisons across diverse content categories, contrasting single-stage input-prompt inference with adaptive two-stage inference under the same prompt.

Wan2.2-5B · Additional Comparisons

Prompt: A red fox leaps over a stream in an autumn birch grove, where fallen orange leaves fly from the bank, wet stones border the water, and thin white trunks stand behind the fox.

Baseline

Ours

Prompt: A clown fish swims among sea anemone tentacles, with orange stripes, small fins, waving anemone arms, and coral textures around it.

Baseline

Ours

Prompt: A tall sunflower with a heavy yellow head slowly turns in an open farm field, with green leaves twisting along the stem and rows of other sunflowers behind it.

Baseline

Ours

Prompt: Orange juice splashes into a tall glass, forming bubbles at the top and droplets on the clear rim.

Baseline

Ours

Prompt: A yellow butterfly flies between flowers in an alpine meadow, where purple lupines sway below it, patches of white snow remain on distant slopes, and grass stems bend in the wind.

Baseline

Ours

Prompt: A red squirrel runs along a log in a temperate forest floor, where moss covers the fallen trunk, acorns scatter near its paws, and ferns grow around the shaded clearing.

Baseline

Ours

Prompt: A baker in a white apron kneads a round dough ball on a floured wooden table, with copper bowls and stacked proofing baskets behind the counter.

Baseline

Ours

Prompt: A scuba diver enters a coral reef, releasing bubbles while yellow reef fish pass between branching coral and a blue coral wall rises nearby.

Baseline

Ours

CogVideoX-5B · Additional Comparisons

Prompt: A seagull walks along the shore, stepping over wet sand while small waves roll behind it and shell fragments dot the beach.

Baseline

Ours

Prompt: A colorful butterfly perches on a flower bud, with patterned wings folded above the petals, green stems around it, and other buds blurred behind.

Baseline

Ours

Prompt: People applaud a kids' performance, clapping in rows of seats while children stand on a small stage under bright lights.

Baseline

Ours

Prompt: White flowers bloom on a tree in close up, with small petals, yellow centers, thin branches, and green leaves behind them.

Baseline

Ours

Prompt: Tree leaves change color during autumn, with green, yellow, and orange leaves mixed across branches and fallen leaves covering the ground.

Baseline

Ours

Prompt: A curious cat sits indoors and looks around, with bright eyes, upright ears, a chair behind it, and light from a window crossing the floor.