You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Ring attention with the default native backend fails with an unhelpful error #15308
With context parallelism, ring_degree > 1 and the default native attention backend fail in the first forward with an error that does not say what to do:
ValueError: Native attention does not support return_lse=True
Ring attention needs the attention LSE, which scaled_dot_product_attention does not return. enable_parallelism accepts the native backend because it supports context parallelism (Ulysses works with it), so the problem only shows up once attention runs. Setting a backend that returns the LSE, such as _native_flash, fixes it. I plan to open a PR that makes the error name the cause and these backends.
Reproduction
importosimporttorchfromtorchimportdistributedasdistfromdiffusersimportContextParallelConfig, DiffusionPipelinedist.init_process_group(backend="nccl")
device=torch.device(f"cuda:{int(os.environ['LOCAL_RANK'])}")
torch.cuda.set_device(device)
pipe=DiffusionPipeline.from_pretrained("Wan-AI/Wan2.2-TI2V-5B-Diffusers", torch_dtype=torch.bfloat16).to(device)
pipe.transformer.enable_parallelism(config=ContextParallelConfig(ring_degree=2))
# pipe.transformer.set_attention_backend("_native_flash") # with this line it runspipe(prompt="A red fox running through a snowy forest", height=480, width=832, num_frames=49, num_inference_steps=8)
dist.destroy_process_group()
torchrun --nproc-per-node 2 repro.py
Logs
ValueError: Native attention does not support return_lse=True
Activity