Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/source/en/api/models/ltx2_diffusion_decoder.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,65 @@ reproduce the untiled result exactly; the default tile and overlap sizes match t
Neighborhood attention rejects any grid smaller than its kernel, so a trailing remnant tile is merged into its
neighbor rather than decoded on its own.

## Keyframe-aware decoding

A decode can be anchored on *keyframe planes*: single-frame latents at known pixel frames, for example the generated
slots of [`LTX2DFRPipeline`]. Each plane must be the latent of a standalone one-frame clip (the VAE is causal, so a
frame encoded inside a clip is a different latent), denormalized like the video latents. Positions are pixel-frame
indices, not latent-frame indices.

Omit both arguments for a plain decode; that path does not depend on whether the checkpoint carries a keyframe tag.

```python
video = decoder.decode(
latents, # (B, C, F, H, W), already denormalized
generator=torch.Generator("cuda").manual_seed(0),
keyframe_latents=keyframe_latents, # (B, C, P, H, W)
keyframe_frame_indices=[24, 48], # pixel frame of each plane
).sample
```

The planes go through the same weights as the video. In every attention layer each video position also attends to the
same spatial window on its two nearest planes (frames before the first plane attend to that plane alone), and each
plane to the same window on its two nearest frames. This joint attention runs on a built-in PyTorch implementation
whatever attention processor is set, since neither FlexAttention's block mask nor NATTEN expresses it.

Checkpoints trained for keyframe decoding also carry a learned tag added to the plane latents, which
`decoder_keyframe_type_embedding=True` creates as `decoder.type_emb`. Decoding with keyframe planes on a checkpoint that
lacks that tag raises rather than decoding the planes untagged. If a checkpoint's weights carry `decoder.type_emb` but
its config does not set the flag, pass `decoder_keyframe_type_embedding=True` to
[`LTX2VideoDiffusionDecoderModel.from_pretrained`]; otherwise the tag is dropped as an unexpected key. Converting the
original VAE with `convert_ltx2_diffusion_video_vae` sets the flag when the tag is present.

With tiling enabled each temporal tile keeps the planes inside it plus the nearest plane on each side.

### Standard pipeline

[`LTX2VideoDiffusionDecodePipeline`] forwards the same arguments. A DFR pass with `output_type="latent"` already
denormalized both the video and the slots:

```python
from diffusers import LTX2DFRPipeline, LTX2VideoDiffusionDecodePipeline, LTX2VideoDiffusionDecoderModel

dfr = LTX2DFRPipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16)
out = dfr(prompt="...", output_type="latent", generate_slots=True)

decoder = LTX2VideoDiffusionDecoderModel.from_pretrained(
"Lightricks/LTX-2.5-Diffusers",
subfolder="diffusion_decoder",
dtype=torch.bfloat16,
decoder_keyframe_type_embedding=True, # keep the trained keyframe tag `decoder.type_emb`
)
decode_pipe = LTX2VideoDiffusionDecodePipeline(diffusion_decoder=decoder, scheduler=dfr.scheduler)
video = decode_pipe(
out.frames,
generator=torch.Generator("cuda").manual_seed(0),
denormalize=False,
keyframe_latents=out.keyframes,
keyframe_frame_indices=out.keyframe_positions,
).frames
```

## LTX2VideoDiffusionDecoderModel

[[autodoc]] LTX2VideoDiffusionDecoderModel
Expand Down
16 changes: 13 additions & 3 deletions docs/source/en/api/pipelines/ltx2.md
Original file line number Diff line number Diff line change
Expand Up @@ -1311,21 +1311,31 @@ pipe.transformer.disable_adapters()

The two axes are seamed differently. Neither side of a spatial border holds a known answer, so those overlaps are blended with trapezoidal weights. Temporal tiles are cut on the last refine round's keyframe seams.

**Decoding with the diffusion decoder.** For maximum detail fidelity, stay on `output_type="latent"` and hand the (already denormalized, possibly `trim_canvas`'d) latents to [`LTX2VideoDiffusionDecodePipeline`].
**Decoding with the diffusion decoder.** For maximum detail fidelity, stay on `output_type="latent"` and hand the (already denormalized, possibly `trim_canvas`'d) latents to [`LTX2VideoDiffusionDecodePipeline`]. When the pass produced keyframe slots, pass those too so the decoder can attend to them:

```py
from diffusers import LTX2VideoDiffusionDecodePipeline
from diffusers.models.autoencoders.ltx2_diffusion_decoder import LTX2VideoDiffusionDecoderModel

decoder = LTX2VideoDiffusionDecoderModel.from_pretrained(
"Lightricks/LTX-2.5-Diffusers", subfolder="diffusion_decoder", dtype=torch.bfloat16
"Lightricks/LTX-2.5-Diffusers",
subfolder="diffusion_decoder",
dtype=torch.bfloat16,
decoder_keyframe_type_embedding=True, # keep the trained keyframe tag `decoder.type_emb`
)
decode_pipe = LTX2VideoDiffusionDecodePipeline(
diffusion_decoder=decoder, scheduler=pipe.scheduler, vae=pipe.vae
)
decode_pipe.enable_model_cpu_offload()
# `denormalize=False`: the `output_type="latent"` path already applied the latent statistics.
video = decode_pipe(latents=out3.frames, denormalize=False, output_type="np", return_dict=False)[0]
video = decode_pipe(
latents=out3.frames,
denormalize=False,
keyframe_latents=out3.keyframes,
keyframe_frame_indices=out3.keyframe_positions,
output_type="np",
return_dict=False,
)[0]
```

## LTX2Pipeline
Expand Down
4 changes: 4 additions & 0 deletions scripts/convert_ltx2_to_diffusers.py
Original file line number Diff line number Diff line change
Expand Up @@ -895,6 +895,10 @@ def get_ltx2_diffusion_video_vae_config(version: str) -> tuple[dict[str, Any], d
def convert_ltx2_diffusion_video_vae(original_state_dict: dict[str, Any], version: str) -> dict[str, Any]:
config, rename_dict, special_keys_remap = get_ltx2_diffusion_video_vae_config(version)
diffusers_config = config["diffusers_config"]
# Checkpoints trained for keyframe-aware decoding carry the keyframe stream's learned tag, `decoder.type_emb`
# (`(latent_channels,)`); older ones do not. It keeps its name, so only the config has to know it is there.
if "decoder.type_emb" in original_state_dict:
diffusers_config = {**diffusers_config, "decoder_keyframe_type_embedding": True}

with init_empty_weights():
vae = LTX2VideoDiffusionDecoderModel.from_config(diffusers_config)
Expand Down
Loading