Repository navigation
Description
Describe the bug
With group offloading to disk (offload_to_disk_path), buffers that a pipeline reads directly, without calling the owning module's forward, hold uninitialized values. Flux2KleinPipeline reads vae.bn.running_mean / vae.bn.running_var to scale the latents, so with the VAE group-offloaded to disk the output image is all black. With CPU-memory offloading the same repro gives an identical image.
_offload_to_disk replaces every offloaded tensor with torch.empty_like(...). That is fine for tensors the hooks onload before use, but vae.bn's forward is never called, so its buffers are never onloaded and the pipeline reads the uninitialized CPU tensors. Memory offloading keeps the values (it moves the tensors to the CPU, or restores pinned CPU copies with streams). The same direct reads are in pipeline_flux2.py, pipeline_flux2_klein*.py, pipeline_ernie_image.py, pipeline_bria_fibo2.py, pipeline_ideogram4.py and the flux2 / flux3 / ernie_image / ideogram4 modular blocks.
I have a fix (keep buffer values on the offload device in _offload_to_disk, as memory offloading does) with a regression test and can open a PR.
Reproduction
import tempfile
import numpy as np
import torch
from diffusers import Flux2KleinPipeline
pipe = Flux2KleinPipeline.from_pretrained("black-forest-labs/FLUX.2-klein-4B", torch_dtype=torch.bfloat16)
pipe.to("cuda")
kwargs = dict(
prompt="A cat holding a sign that says hello world",
height=1024,
width=1024,
guidance_scale=1.0,
num_inference_steps=4,
)
ref = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(0)).images[0]
running_mean = pipe.vae.bn.running_mean.clone()
pipe.vae.to("cpu")
pipe.vae.enable_group_offload(
onload_device=torch.device("cuda"),
offload_device=torch.device("cpu"),
offload_type="leaf_level",
offload_to_disk_path=tempfile.mkdtemp(),
)
out = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(0)).images[0]
print("bn.running_mean unchanged:", torch.equal(pipe.vae.bn.running_mean.to("cuda"), running_mean))
diff = np.abs(np.asarray(ref, dtype=np.int16) - np.asarray(out, dtype=np.int16))
print(f"image vs. no offload: max abs diff {diff.max()}, mean abs diff {diff.mean():.2f}")
ref.save("ref.png")
out.save("vae_disk_offload.png")Logs
bn.running_mean unchanged: False
image vs. no offload: max abs diff 255, mean abs diff 149.37vae_disk_offload.png is all black. With the fix the output is identical to ref.png (max abs diff 0). Checking only the bn buffers after enable_group_offload on main: leaf_level and block_level, with and without use_stream, are all wrong with offload_to_disk_path and all correct with CPU-memory offloading.
System Info
- Diffusers version: 0.44.0.dev0 (main, e84bda1)
- Platform: Linux-7.0.0-42-generic-x86_64-with-glibc2.39
- Python version: 3.12.10
- PyTorch version (GPU?): 2.16.0a0+rocm10.3.0a20261115 (True)
- Huggingface_hub version: 1.33.0
- Transformers version: 5.21.0
- Accelerate version: 1.9.0
- Safetensors version: 0.8.0
- GPU: AMD Radeon AI PRO R9700
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
Who can help?
No response
Activity