Skip to content

ControlNet union pipeline fails on multi-model #10656

Closed
@vladmandic

Description

@vladmandic

Describe the bug

All controlnet types are typically defined inside pipeline as below (example from StableDiffusionXLControlNetPipeline):

controlnet: Union[ControlNetModel, List[ControlNetModel], Tuple[ControlNetModel], MultiControlNetModel],

however, StableDiffusionXLControlNetUnionPipeline pipeline defines it simply as:

controlnet: ControlNetUnionModel

which defeats one of the main advantages of union controlnet - to be able to perform multiple guidances using same model.
for reference, controlnetunion was added via pr #10131
any changes to txt2img pipeline should also be mirrored in img2img and inpaint pipelines.

Reproduction

control1 = ControlNetUnionModel.from_single_file(...)
control2 = ControlNetUnionModel.from_single_file(...)
pipe = StableDiffusionXLControlNetUnionPipeline.from_single_file(..., control=[control1, control2])

Logs

│    256 │   │   if not isinstance(controlnet, ControlNetUnionModel):                                                                                                                                                                                                                                                                                                                                                         │
│ ❱  257 │   │   │   raise ValueError("Expected `controlnet` to be of type `ControlNetUnionModel`.")                                                                                                                                                                                                                                                                                                                          │
│    258                                                                                                                                                                                                                                                                                                                                                                                                                      │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Expected `controlnet` to be of type `ControlNetUnionModel`.

System Info

diffusers==0.33.0.dev0

Who can help?

@hlky @yiyixuxu @sayakpaul @DN6

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleIssues that haven't received updates

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions