Skip to content

Ring attention with the default native backend fails with an unhelpful error #15308

Closed as not planned

Description

@klin2024

Describe the bug

With context parallelism, ring_degree > 1 and the default native attention backend fail in the first forward with an error that does not say what to do:

ValueError: Native attention does not support return_lse=True

Ring attention needs the attention LSE, which scaled_dot_product_attention does not return. enable_parallelism accepts the native backend because it supports context parallelism (Ulysses works with it), so the problem only shows up once attention runs. Setting a backend that returns the LSE, such as _native_flash, fixes it. I plan to open a PR that makes the error name the cause and these backends.

Reproduction

import os

import torch
from torch import distributed as dist

from diffusers import ContextParallelConfig, DiffusionPipeline

dist.init_process_group(backend="nccl")
device = torch.device(f"cuda:{int(os.environ['LOCAL_RANK'])}")
torch.cuda.set_device(device)

pipe = DiffusionPipeline.from_pretrained("Wan-AI/Wan2.2-TI2V-5B-Diffusers", torch_dtype=torch.bfloat16).to(device)
pipe.transformer.enable_parallelism(config=ContextParallelConfig(ring_degree=2))
# pipe.transformer.set_attention_backend("_native_flash")  # with this line it runs
pipe(prompt="A red fox running through a snowy forest", height=480, width=832, num_frames=49, num_inference_steps=8)
dist.destroy_process_group()

torchrun --nproc-per-node 2 repro.py

Logs

ValueError: Native attention does not support return_lse=True

System Info

  • Diffusers version: 0.44.0.dev0 (main, 1dabc29)
  • Platform: Linux-6.8.0-138-generic-x86_64-with-glibc2.35
  • Python version: 3.12.4
  • PyTorch version (GPU?): 2.14.1+cu130 (True)
  • Transformers version: 5.23.0
  • Accelerator: 2x NVIDIA H100 80GB HBM3
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: yes, context parallelism with ring_degree=2 via torchrun

Who can help?

@sayakpaul @stevhliu

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions