Skip to content

Group offloading to disk leaves buffers read without a forward call uninitialized (FLUX.2 Klein all-black image) #15332

Open

Description

@klin2024

Describe the bug

With group offloading to disk (offload_to_disk_path), buffers that a pipeline reads directly, without calling the owning module's forward, hold uninitialized values. Flux2KleinPipeline reads vae.bn.running_mean / vae.bn.running_var to scale the latents, so with the VAE group-offloaded to disk the output image is all black. With CPU-memory offloading the same repro gives an identical image.

_offload_to_disk replaces every offloaded tensor with torch.empty_like(...). That is fine for tensors the hooks onload before use, but vae.bn's forward is never called, so its buffers are never onloaded and the pipeline reads the uninitialized CPU tensors. Memory offloading keeps the values (it moves the tensors to the CPU, or restores pinned CPU copies with streams). The same direct reads are in pipeline_flux2.py, pipeline_flux2_klein*.py, pipeline_ernie_image.py, pipeline_bria_fibo2.py, pipeline_ideogram4.py and the flux2 / flux3 / ernie_image / ideogram4 modular blocks.

I have a fix (keep buffer values on the offload device in _offload_to_disk, as memory offloading does) with a regression test and can open a PR.

Reproduction

import tempfile
import numpy as np
import torch
from diffusers import Flux2KleinPipeline

pipe = Flux2KleinPipeline.from_pretrained("black-forest-labs/FLUX.2-klein-4B", torch_dtype=torch.bfloat16)
pipe.to("cuda")
kwargs = dict(
    prompt="A cat holding a sign that says hello world",
    height=1024,
    width=1024,
    guidance_scale=1.0,
    num_inference_steps=4,
)
ref = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(0)).images[0]

running_mean = pipe.vae.bn.running_mean.clone()
pipe.vae.to("cpu")
pipe.vae.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="leaf_level",
    offload_to_disk_path=tempfile.mkdtemp(),
)
out = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(0)).images[0]

print("bn.running_mean unchanged:", torch.equal(pipe.vae.bn.running_mean.to("cuda"), running_mean))
diff = np.abs(np.asarray(ref, dtype=np.int16) - np.asarray(out, dtype=np.int16))
print(f"image vs. no offload: max abs diff {diff.max()}, mean abs diff {diff.mean():.2f}")
ref.save("ref.png")
out.save("vae_disk_offload.png")

Logs

bn.running_mean unchanged: False
image vs. no offload: max abs diff 255, mean abs diff 149.37

vae_disk_offload.png is all black. With the fix the output is identical to ref.png (max abs diff 0). Checking only the bn buffers after enable_group_offload on main: leaf_level and block_level, with and without use_stream, are all wrong with offload_to_disk_path and all correct with CPU-memory offloading.

System Info

  • Diffusers version: 0.44.0.dev0 (main, e84bda1)
  • Platform: Linux-7.0.0-42-generic-x86_64-with-glibc2.39
  • Python version: 3.12.10
  • PyTorch version (GPU?): 2.16.0a0+rocm10.3.0a20261115 (True)
  • Huggingface_hub version: 1.33.0
  • Transformers version: 5.21.0
  • Accelerate version: 1.9.0
  • Safetensors version: 0.8.0
  • GPU: AMD Radeon AI PRO R9700
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: no

Who can help?

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions