Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions examples/instruct_pix2pix/train_instruct_pix2pix_sdxl.py
Original file line number Diff line number Diff line change
Expand Up @@ -872,15 +872,14 @@ def compute_embeddings_for_prompts(prompts, text_encoders, tokenizers):
def compute_null_conditioning():
null_conditioning_list = []
for a_tokenizer, a_text_encoder in zip(tokenizers, text_encoders):
null_conditioning_list.append(
a_text_encoder(
tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device),
output_hidden_states=True,
).hidden_states[-2]
null_encoder_output = a_text_encoder(
tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device),
output_hidden_states=True,
)
return torch.concat(null_conditioning_list, dim=-1)
null_conditioning_list.append(null_encoder_output.hidden_states[-2])
return torch.concat(null_conditioning_list, dim=-1), null_encoder_output[0]
Comment on lines -876 to +880

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why this change is needed?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script already encodes the empty prompt, but it only keeps the token embeddings. I’m also keeping the pooled embedding from the second text encoder so we can use it when we drop a prompt. This uses the same encoder pass.


null_conditioning = compute_null_conditioning()
null_conditioning, null_pooled_text_embeds = compute_null_conditioning()

def compute_time_ids():
crops_coords_top_left = (args.crops_coords_top_left_h, args.crops_coords_top_left_w)
Expand Down Expand Up @@ -1080,6 +1079,8 @@ def collate_fn(examples):
prompt_mask = prompt_mask.reshape(bsz, 1, 1)
# Final text conditioning.
encoder_hidden_states = torch.where(prompt_mask, null_conditioning, encoder_hidden_states)
pooled_prompt_mask = prompt_mask.reshape(bsz, 1)
add_text_embeds = torch.where(pooled_prompt_mask, null_pooled_text_embeds, add_text_embeds)
Comment on lines +1082 to +1083

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need it?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a prompt is dropped, the UNet still receives its pooled embedding through add_text_embeds. I ran one training step twice with all prompts dropped, changing only the prompt text. Before the fix, the two runs produced different UNet weights. With the same mask applied to add_text_embeds, the weights matched exactly.


# Sample masks for the original images.
image_mask_dtype = original_image_embeds.dtype
Expand Down