Skip to content

Fix pooled text conditioning dropout in SDXL InstructPix2Pix training - #15247

Open
mzeynali wants to merge 2 commits into
huggingface:mainfrom
mzeynali:fix/sdxl-instruct-pix2pix-conditioning-dropout
Open

mzeynali wants to merge 2 commits into
huggingface:mainfrom
mzeynali:fix/sdxl-instruct-pix2pix-conditioning-dropout

Conversation

@mzeynali

Copy link
Copy Markdown

What does this PR do?

Fixes text conditioning leakage in SDXL InstructPix2Pix training. When a prompt was dropped, encoder_hidden_states was replaced with empty-prompt conditioning, but add_text_embeds still contained the original prompt's pooled embedding.

This change also replaces the pooled embedding with the second text encoder's empty-prompt output, using the same dropout mask.

Validation

Added a regression test using tiny, real SDXL components and local synthetic data:

  • With full text dropout, changing only the prompt produces identical trained UNet weights.
  • Without dropout, changing the prompt affects the trained weights.

Both cases pass with the fix. The full-dropout case fails on the original code.

Lint, formatting, and check_ai.py passed. make style and make fix-copies passed in an isolated snapshot. GPU training was not tested.

AI-assisted self-review

Reviewed against the repository guidelines. No blocking findings or dead code were identified, and no findings were intentionally left unresolved.

@github-actions github-actions Bot added examples size/M PR with diff < 200 LOC labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Hi @mzeynali, thanks for the PR! It does not appear to link an issue it fixes.

  • If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked.
  • If you are the author of the model/pipeline this PR relates to, or part of the team we work with on it, tick the Are you the author (or part of the team we work with) of the model/pipeline this PR relates to? box in the PR description (if you removed the template, paste that line back, checked). This reminder then no longer applies.
  • Otherwise there is nothing you need to do: a maintainer will look at the PR and add the no-issue-needed label to silence this reminder if it should stay open without an issue.

See the contribution guide for more details.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an existing issue, the author box is ticked, or the no-issue-needed label is added, you can ignore this message — it stays here as a comment, but it no longer applies.

@sayakpaul sayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have asked some questions. Please provide simple answers in your own words.

@@ -0,0 +1,134 @@
# coding=utf-8

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I’ll remove the test file from this PR.

Comment on lines -876 to +880
a_text_encoder(
tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device),
output_hidden_states=True,
).hidden_states[-2]
null_encoder_output = a_text_encoder(
tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device),
output_hidden_states=True,
)
return torch.concat(null_conditioning_list, dim=-1)
null_conditioning_list.append(null_encoder_output.hidden_states[-2])
return torch.concat(null_conditioning_list, dim=-1), null_encoder_output[0]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why this change is needed?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script already encodes the empty prompt, but it only keeps the token embeddings. I’m also keeping the pooled embedding from the second text encoder so we can use it when we drop a prompt. This uses the same encoder pass.

Comment on lines +1082 to +1083
pooled_prompt_mask = prompt_mask.reshape(bsz, 1)
add_text_embeds = torch.where(pooled_prompt_mask, null_pooled_text_embeds, add_text_embeds)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need it?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a prompt is dropped, the UNet still receives its pooled embedding through add_text_embeds. I ran one training step twice with all prompts dropped, changing only the prompt text. Before the fix, the two runs produced different UNet weights. With the same mask applied to add_text_embeds, the weights matched exactly.

@github-actions github-actions Bot added size/S PR with diff < 50 LOC and removed size/M PR with diff < 200 LOC labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples size/S PR with diff < 50 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants