Repository navigation
Fix pooled text conditioning dropout in SDXL InstructPix2Pix training #15247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -872,15 +872,14 @@ def compute_embeddings_for_prompts(prompts, text_encoders, tokenizers): | |
| def compute_null_conditioning(): | ||
| null_conditioning_list = [] | ||
| for a_tokenizer, a_text_encoder in zip(tokenizers, text_encoders): | ||
| null_conditioning_list.append( | ||
| a_text_encoder( | ||
| tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device), | ||
| output_hidden_states=True, | ||
| ).hidden_states[-2] | ||
| null_encoder_output = a_text_encoder( | ||
| tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device), | ||
| output_hidden_states=True, | ||
| ) | ||
| return torch.concat(null_conditioning_list, dim=-1) | ||
| null_conditioning_list.append(null_encoder_output.hidden_states[-2]) | ||
| return torch.concat(null_conditioning_list, dim=-1), null_encoder_output[0] | ||
|
|
||
| null_conditioning = compute_null_conditioning() | ||
| null_conditioning, null_pooled_text_embeds = compute_null_conditioning() | ||
|
|
||
| def compute_time_ids(): | ||
| crops_coords_top_left = (args.crops_coords_top_left_h, args.crops_coords_top_left_w) | ||
|
|
@@ -1080,6 +1079,8 @@ def collate_fn(examples): | |
| prompt_mask = prompt_mask.reshape(bsz, 1, 1) | ||
| # Final text conditioning. | ||
| encoder_hidden_states = torch.where(prompt_mask, null_conditioning, encoder_hidden_states) | ||
| pooled_prompt_mask = prompt_mask.reshape(bsz, 1) | ||
| add_text_embeds = torch.where(pooled_prompt_mask, null_pooled_text_embeds, add_text_embeds) | ||
|
Comment on lines
+1082
to
+1083
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we need it?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When a prompt is dropped, the UNet still receives its pooled embedding through |
||
|
|
||
| # Sample masks for the original images. | ||
| image_mask_dtype = original_image_embeds.dtype | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you explain why this change is needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The script already encodes the empty prompt, but it only keeps the token embeddings. I’m also keeping the pooled embedding from the second text encoder so we can use it when we drop a prompt. This uses the same encoder pass.