Repository navigation
Description
Describe the bug
KolorsPAGPipeline (pipeline_pag_kolors.py) fails when IP-Adapter inputs are passed. The result of prepare_ip_adapter_image_embeds() is assigned to image_embeds instead of ip_adapter_image_embeds:
- with
ip_adapter_image, the following loop iteratesip_adapter_image_embeds, which is stillNone->TypeError; - with
ip_adapter_image_embeds, the UNet getsadded_cond_kwargs["image_embeds"] = image_embeds, i.e. only the last tensor of that loop instead of the list. With two IP-Adapters this raisesValueError; with one it runs, but through the deprecated "tensor instead of list" fallback (FutureWarning).
The block was added in #8934: it took the PAG loop over ip_adapter_image_embeds from StableDiffusionXLPAGPipeline but kept KolorsPipeline's image_embeds variable (KolorsPipeline has no such loop, so it works there). The fix is to do what StableDiffusionXLPAGPipeline does: assign to ip_adapter_image_embeds and pass that list to the UNet. I have the fix with an IPAdapterTesterMixin test class for Kolors PAG and am happy to open a PR.
Reproduction
Tiny random components and two IP-Adapters with random weights; runs on CPU:
import numpy as np
import torch
from PIL import Image
from transformers import CLIPImageProcessor, CLIPVisionConfig, CLIPVisionModelWithProjection
from diffusers import AutoencoderKL, EulerDiscreteScheduler, KolorsPAGPipeline, UNet2DConditionModel
from diffusers.pipelines.kolors import ChatGLMModel, ChatGLMTokenizer
def tiny_unet():
# Tiny random Kolors UNet (same layout as tests/pipelines/pag/test_pag_kolors.py)
torch.manual_seed(0)
return UNet2DConditionModel(
block_out_channels=(2, 4),
layers_per_block=2,
sample_size=32,
in_channels=4,
out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4),
use_linear_projection=True,
addition_embed_type="text_time",
addition_time_embed_dim=8,
transformer_layers_per_block=(1, 2),
projection_class_embeddings_input_dim=56,
cross_attention_dim=8,
norm_num_groups=1,
)
def tiny_pipe():
return KolorsPAGPipeline(
vae=AutoencoderKL(
block_out_channels=[32, 64],
down_block_types=["DownEncoderBlock2D"] * 2,
up_block_types=["UpDecoderBlock2D"] * 2,
latent_channels=4,
),
text_encoder=ChatGLMModel.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b", dtype=torch.float32),
tokenizer=ChatGLMTokenizer.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b"),
unet=tiny_unet(),
scheduler=EulerDiscreteScheduler(),
image_encoder=CLIPVisionModelWithProjection(
CLIPVisionConfig(
hidden_size=8,
intermediate_size=16,
num_attention_heads=2,
num_hidden_layers=1,
image_size=32,
patch_size=8,
projection_dim=8,
)
),
feature_extractor=CLIPImageProcessor(size=32, crop_size=32),
pag_applied_layers=["mid"],
)
# Two random IP-Adapters in the original IP-Adapter layout
pipe = tiny_pipe()
dim = pipe.unet.config.cross_attention_dim
ip_layers, key_id = {}, 1
for name in pipe.unet.attn_processors:
if name.endswith("attn1.processor"):
continue
if name.startswith("mid_block"):
hidden = pipe.unet.config.block_out_channels[-1]
elif name.startswith("up_blocks"):
hidden = pipe.unet.config.block_out_channels[::-1][int(name[len("up_blocks.")])]
else:
hidden = pipe.unet.config.block_out_channels[int(name[len("down_blocks.")])]
ip_layers[f"{key_id}.to_k_ip.weight"] = torch.randn(hidden, dim)
ip_layers[f"{key_id}.to_v_ip.weight"] = torch.randn(hidden, dim)
key_id += 2
image_proj = {
"proj.weight": torch.randn(4 * dim, dim),
"proj.bias": torch.zeros(4 * dim),
"norm.weight": torch.ones(dim),
"norm.bias": torch.zeros(dim),
}
ip_adapter = {"image_proj": image_proj, "ip_adapter": ip_layers}
pipe.load_ip_adapter([ip_adapter, ip_adapter], subfolder=["", ""], weight_name=["", ""], image_encoder_folder=None)
pipe.set_ip_adapter_scale([0.6, 0.6])
ip_inputs = {
"ip_adapter_image": {"ip_adapter_image": [Image.fromarray(np.zeros((32, 32, 3), dtype=np.uint8))] * 2},
"ip_adapter_image_embeds": {"ip_adapter_image_embeds": [torch.randn(2, 1, dim)] * 2}, # (negative, positive)
}
for name, kwargs in ip_inputs.items():
try:
image = pipe(
"a photo of a cat",
guidance_scale=5.0,
pag_scale=3.0,
num_inference_steps=2,
height=64,
width=64,
output_type="np",
**kwargs,
).images
print(f"{name}: OK, output {image.shape}, finite {bool(np.isfinite(image).all())}")
except Exception as e:
print(f"{name}: {type(e).__name__}: {e}")Context
I was running documented features (PAG, IP-Adapter, LoRA) across the Kolors pipelines on current main, looking for bugs in the library with an AI agent (Claude Code). The script above was run as shown, on main (2bb9468) and with the fix, on a machine with an RTX 5090 (the script itself runs on CPU), with PYTHONNOUSERSITE=1 and HF_HOME set to a separate cache. The logs below are its output on main; with the fix both lines print OK.
The same failure happens with the real weights on the RTX 5090 (Kwai-Kolors/Kolors-diffusers fp16, 1024x1024, pag_applied_layers=["mid"], pag_scale=1.5): the IP-Adapter example from the Kolors docs, run with KolorsPAGPipeline, fails with the same TypeError.
Logs
ip_adapter_image: TypeError: 'NoneType' object is not iterable
ip_adapter_image_embeds: ValueError: image_embeds must have the same length as image_projection_layers, got 1 and 2System Info
- 🤗 Diffusers version: 0.44.0.dev0
- Platform: Windows-11-10.0.26200-SP0
- Running on Google Colab?: No
- Python version: 3.12.16
- PyTorch version (GPU?): 2.14.1+cu130 (True)
- Huggingface_hub version: 1.32.0
- Transformers version: 5.16.1
- Accelerate version: 1.9.0
- PEFT version: 0.21.3
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 5090, 32607 MiB
- Using GPU in script?: No (the script runs on CPU)
- Using distributed or parallel set-up in script?: No
Activity