Reducing model loads on pipeline switches

[pdoane]

What API design would you like to have changed or added to the library? Why?

When switching between txt2img and img2img, it is possible to use the constructor directly:

StableDiffusionImg2ImgPipeline(**prev_pipeline.pipe.components)

I would like to enable similar patterns when using ControlNet. There are potentially 3 different scenarios here:

1. Same pipeline but different controlnet. It doesn't look like there is anything in the constructor prohibiting this but I have hit some issues - possibly bugs in my own code.

2. Different pipeline but same base model. I get some errors about unallocated storage when testing this.

3. Different controlnet pipeline but same base model. This also seems to sometimes work but I've hit some issues.

What use case would this enable or better enable? Can you give us a code example?

Model loading is slow and to a lesser extent, the control net models. I would like to minimize the loading times and re-use existing components that are in memory.

[pdoane]

This was a lack of understanding about how to and similar methods on a pipeline work. I was unaware about the module registration and how many pipeline methods are broadcasting parameters to their registered modules. An example of the problem:

pipe = StableDiffusionPipeline.from_pretrained(...)
pipe.to('mps')
controlnet = ControlNetModel.from_pretrained(...)
pipe2 = StableDiffusionControlNetPipeline(**pipe.components, controlnet=controlnet)
pipe2(...)

In this case pipe2 is on the mps device, but controlnet is on the CPU. Adding an explicit to call on controlnet (as well as other pipeline calls that route to modules) looks like it fixes things.

[philippwulff]

Hi @pdoane

How did you do it with StableDiffusionImg2ImgPipeline?

Following the example for vanilla stable diffusion like this does not work:

                self.text2img = StableDiffusionControlNetPipeline.from_pretrained(MODEL_PATH, controlnet=controlnet, torch_dtype=torch.float16).to(device)
                self.img2img = StableDiffusionControlNetImg2ImgPipeline(**self.text2img.components).to(device)
                self.inpaint = StableDiffusionControlNetInpaintPipeline(**self.text2img.components).to(device)

I also want to switch quickly between inpainting and img2img...

[pdoane]

It's possible newer diffusers versions of changed the API (sorry I haven't updated in awhile), but you can see how I abstracted this here:

https://github.com/pdoane/seed-alchemy/blob/main/backend/universal_pipeline.py

I use StableDiffusionPipeline as the base, which allows switching controlnet models without reloading the SD model. This requires some additional management on the controlnet object:

                control_net.to(device)
                control_net.set_attention_slice("auto")

What error are you hitting?