Skip to content

fix compatability issue between PAG and IP-adapter #8379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 5, 2024
Merged

fix compatability issue between PAG and IP-adapter #8379

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0ddc76a
fix compatability issue between PAG and IP-adapter
sunovivid Jun 2, 2024
1edb67d
fix compatibility issue between PAG and IP-adapter plus
sunovivid Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions src/diffusers/loaders/unet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -928,9 +928,7 @@ def _convert_ip_adapter_attn_to_diffusers(self, state_dicts, low_cpu_mem_usage=F
hidden_size = self.config.block_out_channels[block_id]

if cross_attention_dim is None or "motion_modules" in name:
attn_processor_class = (
AttnProcessor2_0 if hasattr(F, "scaled_dot_product_attention") else AttnProcessor
)
attn_processor_class = self.attn_processors[name].__class__
attn_procs[name] = attn_processor_class()

else:
Expand Down
5 changes: 3 additions & 2 deletions src/diffusers/pipelines/pag_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def enable_pag(
self._pag_applied_layers = pag_applied_layers
self._pag_applied_layers_index = pag_applied_layers_index
self._pag_cfg = pag_cfg

self._is_pag_enabled = True
self._set_pag_attn_processor()

def _get_self_attn_layers(self):
Expand Down Expand Up @@ -180,6 +180,7 @@ def disable_pag(self):
self._pag_applied_layers = None
self._pag_applied_layers_index = None
self._pag_cfg = None
self._is_pag_enabled = False

@property
def pag_adaptive_scaling(self):
Expand All @@ -191,4 +192,4 @@ def do_pag_adaptive_scaling(self):

@property
def do_perturbed_attention_guidance(self):
return hasattr(self, "_pag_scale") and self._pag_scale is not None and self._pag_scale > 0
return self._is_pag_enabled
6 changes: 5 additions & 1 deletion src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1172,6 +1172,10 @@ def __call__(
self.do_classifier_free_guidance,
)

# expand the image embeddings if we are using perturbed-attention guidance
for i in range(len(image_embeds)):
image_embeds[i] = image_embeds[i].repeat(prompt_embeds.shape[0] // latents.shape[0], 1, 1)
Copy link
Member

@asomoza asomoza Jun 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This throws an error with the PLUS versions of IP Adapters each image_embeds is a 4D tensor.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for finding the error! I found the cause of the error and, thanks to this, came up with a more elegant design. I will upload the revised code with the results soon!


# 8. Denoising loop
num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)

Expand Down Expand Up @@ -1205,7 +1209,7 @@ def __call__(
if self.interrupt:
continue

# expand the latents if we are doing classifier free guidance
# expand the latents if we are doing classifier free guidance, perturbed-attention guidance, or both
latent_model_input = torch.cat([latents] * (prompt_embeds.shape[0] // latents.shape[0]))

latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)
Expand Down