Skip to content

Flux Control and HunyuanVideo LoRA converters swap the scale/shift halves of lora_A in final_layer #15305

Open

Description

@MaiAditya

Describe the bug

final_layer.adaLN_modulation.1 in the original checkpoints orders its output (shift, scale), and diffusers' norm_out.linear orders it (scale, shift), so the converters swap the two halves. For a LoRA only lora_B should be swapped: its rows are the layer's outputs. lora_A is (rank, in), so swapping its halves reorders the rank dimension and the converted delta is wrong.

Two converters in loaders/lora_conversion_utils.py swap both:

  • _convert_bfl_flux_control_lora_to_diffusers: swap_scale_shift is applied for every lora_key, including lora_A (weight and bias).
  • _convert_hunyuan_video_lora_to_diffusers: remap_norm_scale_shift_ swaps every final_layer.adaLN_modulation.1 key, lora_A included.

The kohya Flux converter already swaps only lora_B. This is the same class of bug #15276 fixed for Flux2.

Reproduction

import torch
from diffusers.loaders.lora_conversion_utils import _convert_bfl_flux_control_lora_to_diffusers

RANK, INNER = 4, 3072
torch.manual_seed(0)

class FillingDict(dict):
    # dummy tensors for the keys the converter pops that this repro doesn't care about
    def pop(self, key, *default):
        if key in self:
            return super().pop(key)
        if default:
            return default[0]
        return torch.zeros(21504, RANK) if ".lora_B." in key else torch.zeros(RANK, 8)

lora_A = torch.randn(RANK, INNER)
lora_B = torch.randn(2 * INNER, RANK)  # rows: (shift, scale)
sd = FillingDict({
    "final_layer.adaLN_modulation.1.lora_A.weight": lora_A.clone(),
    "final_layer.adaLN_modulation.1.lora_B.weight": lora_B.clone(),
    "double_blocks.0.img_attn.norm.query_norm.scale": torch.ones(1),
})
out = _convert_bfl_flux_control_lora_to_diffusers(sd)
A = out["transformer.norm_out.linear.lora_A.weight"]
B = out["transformer.norm_out.linear.lora_B.weight"]
shift, scale = (lora_B @ lora_A).chunk(2, dim=0)
expected = torch.cat([scale, shift])
print(torch.allclose(B @ A, expected, atol=1e-4), (B @ A - expected).abs().max().item())
print(torch.allclose(B @ lora_A, expected, atol=1e-4))  # lora_A left as is

Logs

False 39.052650451660156
True

The HunyuanVideo converter gives the same result (max error 17.05 in the equivalent check).

System Info

diffusers main @ 43e3002, torch 2.15.0, Python 3.12, macOS (CPU).

Who can help?

@sayakpaul

I have a fix with tests for both converters (swap only lora_B; one test per converter in the style of test_lora_bfl_final_layer_modulation, both fail on main) and can open a PR if you'd like. It may change the expected slices of the slow Flux Control LoRA integration tests, which I can't regenerate without a GPU.

Activity

  1. sayakpaul commented on Nov 16, 2026

    @sayakpaul
    Member

    Feel free to open a PR. Thanks for reporting!

  2. added this to the Release 0.44.0 milestone on Nov 16, 2026
  3. 1OF1Research commented on Nov 16, 2026

    @1OF1Research

    Independent confirmation on the shipped release — diffusers 0.43.0, torch 2.15.0, CPU, Linux: the reporter's repro runs exactly as written. The converted delta does not match the expected (scale, shift) result (False, max diff 39.0527, the same value in the report), the converted lora_A no longer equals the original, and leaving lora_A unchanged makes B @ A match the expected result exactly (True). So the fault is precisely the lora_A half-swap in _convert_bfl_flux_control_lora_to_diffusers: lora_B's rows are the layer's outputs and need the swap, lora_A's halves are rank components and do not. Happy to test a fix PR against this repro when one opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions