Repository navigation
Conversation
|
Hi @mzeynali, thanks for the PR! It does not appear to link an issue it fixes.
See the contribution guide for more details. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an existing issue, the author box is ticked, or the |
sayakpaul
left a comment
There was a problem hiding this comment.
I have asked some questions. Please provide simple answers in your own words.
| @@ -0,0 +1,134 @@ | |||
| # coding=utf-8 | |||
There was a problem hiding this comment.
Thanks, I’ll remove the test file from this PR.
| a_text_encoder( | ||
| tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device), | ||
| output_hidden_states=True, | ||
| ).hidden_states[-2] | ||
| null_encoder_output = a_text_encoder( | ||
| tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device), | ||
| output_hidden_states=True, | ||
| ) | ||
| return torch.concat(null_conditioning_list, dim=-1) | ||
| null_conditioning_list.append(null_encoder_output.hidden_states[-2]) | ||
| return torch.concat(null_conditioning_list, dim=-1), null_encoder_output[0] |
There was a problem hiding this comment.
Could you explain why this change is needed?
There was a problem hiding this comment.
The script already encodes the empty prompt, but it only keeps the token embeddings. I’m also keeping the pooled embedding from the second text encoder so we can use it when we drop a prompt. This uses the same encoder pass.
| pooled_prompt_mask = prompt_mask.reshape(bsz, 1) | ||
| add_text_embeds = torch.where(pooled_prompt_mask, null_pooled_text_embeds, add_text_embeds) |
There was a problem hiding this comment.
When a prompt is dropped, the UNet still receives its pooled embedding through add_text_embeds. I ran one training step twice with all prompts dropped, changing only the prompt text. Before the fix, the two runs produced different UNet weights. With the same mask applied to add_text_embeds, the weights matched exactly.
What does this PR do?
Fixes text conditioning leakage in SDXL InstructPix2Pix training. When a prompt was dropped,
encoder_hidden_stateswas replaced with empty-prompt conditioning, butadd_text_embedsstill contained the original prompt's pooled embedding.This change also replaces the pooled embedding with the second text encoder's empty-prompt output, using the same dropout mask.
Validation
Added a regression test using tiny, real SDXL components and local synthetic data:
Both cases pass with the fix. The full-dropout case fails on the original code.
Lint, formatting, and
check_ai.pypassed.make styleandmake fix-copiespassed in an isolated snapshot. GPU training was not tested.AI-assisted self-review
Reviewed against the repository guidelines. No blocking findings or dead code were identified, and no findings were intentionally left unresolved.