Skip to content

[Feature] MultiControlNet support for SD3Impainting #11251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[Feature] MultiControlNet support for SD3Impainting #11251

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
adc5a24
update
ishan-modi Apr 9, 2025
a9cc08a
update
ishan-modi Apr 9, 2025
bf14dfc
addressed PR comments
ishan-modi Apr 9, 2025
bef9997
update
ishan-modi Apr 9, 2025
39a2cbf
Merge branch 'main' into fixes-issue-11208
yiyixuxu May 2, 2025
ed5572f
Merge branch 'main' into fixes-issue-11208
ishan-modi May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/diffusers/models/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@

if is_torch_available():
_import_structure["adapter"] = ["MultiAdapter", "T2IAdapter"]
_import_structure["auto_model"] = ["AutoModel"]
_import_structure["autoencoders.autoencoder_asym_kl"] = ["AsymmetricAutoencoderKL"]
_import_structure["autoencoders.autoencoder_dc"] = ["AutoencoderDC"]
_import_structure["autoencoders.autoencoder_kl"] = ["AutoencoderKL"]
Expand All @@ -41,7 +42,6 @@
_import_structure["autoencoders.autoencoder_tiny"] = ["AutoencoderTiny"]
_import_structure["autoencoders.consistency_decoder_vae"] = ["ConsistencyDecoderVAE"]
_import_structure["autoencoders.vq_model"] = ["VQModel"]
_import_structure["auto_model"] = ["AutoModel"]
_import_structure["cache_utils"] = ["CacheMixin"]
_import_structure["controlnets.controlnet"] = ["ControlNetModel"]
_import_structure["controlnets.controlnet_flux"] = ["FluxControlNetModel", "FluxMultiControlNetModel"]
Expand Down
23 changes: 21 additions & 2 deletions src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,8 @@ def __init__(
feature_extractor: Optional[SiglipImageProcessor] = None,
):
super().__init__()
if isinstance(controlnet, (list, tuple)):
controlnet = SD3MultiControlNetModel(controlnet)

self.register_modules(
vae=vae,
Expand Down Expand Up @@ -1119,9 +1121,26 @@ def __call__(
width = latent_width * self.vae_scale_factor

elif isinstance(self.controlnet, SD3MultiControlNetModel):
raise NotImplementedError("MultiControlNetModel is not supported for SD3ControlNetInpaintingPipeline.")
control_images = []

for control_image_ in control_image:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fail if control_image is not passed as a list. Looks like SD3 ControlNet and Flux ControlNet pipelines are missing checks for this. See ControlNet Union XL as an example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, it should fail, no?

if isinstance(self.controlnet, SD3ControlNetModel): this statement is a single controlnet model and expects a single image
elif isinstance(self.controlnet, SD3MultiControlNetModel): this statement is multiple controlnet stacked and expects list of images one image per controlnet

let me know if I am missing something ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is expected to fail, but gracefully.

expects list of images one image per controlnet

These checks are missing.

See ControlNet Union XL as an example.

control_image_ = self.prepare_image_with_mask(
image=control_image_,
mask=control_mask,
width=width,
height=height,
batch_size=batch_size * num_images_per_prompt,
num_images_per_prompt=num_images_per_prompt,
device=device,
dtype=dtype,
do_classifier_free_guidance=self.do_classifier_free_guidance,
guess_mode=False,
)
control_images.append(control_image_)

control_image = control_images
else:
assert False
assert ValueError("Controlnet not found. Please check the controlnet model.")

if controlnet_pooled_projections is None:
controlnet_pooled_projections = torch.zeros_like(pooled_prompt_embeds)
Expand Down