Add controlnet and vae from single file

[patrickvonplaten]

What does this PR do?

Working on solving the following issue (+ all the linked ones):
#3907

ControlNet:

from diffusers import StableDiffusionControlnetPipeline, ControlNetModel

url = "https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_canny.pth"  # can also be a local path
model = ControlNetModel.from_single_file(url)

url = "https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/v1-5-pruned.safetensors" # can also be a local path
pipe = StableDiffusionControlnetPipeline.from_single_file(url, controlnet=controlnet)

StableDiffusionControlnetImg2ImgPipeline and StableDiffusionControlnetIinpaintPipeline work fine as well

VAE

from diffusers import AutoencoderKL

url = "https://huggingface.co/stabilityai/sd-vae-ft-mse-original/blob/main/vae-ft-mse-840000-ema-pruned.safetensors"  # can also be local file
model = AutoencoderKL.from_single_file(url)

Also cc @vladmandic

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[EmilioNicolas]

Looking forward to the merge! Thanks for the PR @patrickvonplaten

[sayakpaul]

Looks very nice.

Just wondering if it makes sense to make the passing of the checkpoint names a little less verbose. WDYT about the following?

model = ControlNetModel.from_single_file("lllyasviel/ControlNet-v1-1", filename="control_v11p_sd15_canny.pth")

Not too strongly opinionated about it either.

[yiyixuxu]

awesome!

[hzhou17]

Thank you to everyone that has been working on this. But I still get this error:

"An error occurred during inpainting: type object 'StableDiffusionControlNetInpaintPipeline' has no attribute 'from_single_file"

diffusers version: 0.18.2

Is there something else that I need to update?

[sayakpaul]

@hzhou17 I think you need to install from the branch being used in this PR:

!pip install git+https://github.com/huggingface/diffusers@add_from_single_file_controlnet_vae -q

Here's a Colab Notebook that successfully loads the checkpoints from single files into a StableDiffusionControlNetInpaintPipeline: https://colab.research.google.com/gist/sayakpaul/f99fd09d7817a6189bcc7b54b2757124/scratchpad.ipynb

[patrickvonplaten]

Looks very nice.

Just wondering if it makes sense to make the passing of the checkpoint names a little less verbose. WDYT about the following?

model = ControlNetModel.from_single_file("lllyasviel/ControlNet-v1-1", filename="control_v11p_sd15_canny.pth")

Not too strongly opinionated about it either.

Yes I understand this 100% and thought about it as well - a couple of problems which made me go the current way:

• 1.) Ideally we want the logic for both local file and url. When loading a local file - it really doesn't make too much sense to have an additional filename
• 2.) The "filename" design gets tricky when we have nested filenames as is done here: https://huggingface.co/WarriorMama777/OrangeMixs/tree/main/Models/AbyssOrangeMix2

[sayakpaul]

@patrickvonplaten feel free to merge whenever :) Thanks for explaining #4084 (comment).

[zxf2012]

thanks so much

[williamberman]

iamwavecut#1 this might have introduced a bug @patrickvonplaten

[patrickvonplaten]

It did indeed haha. Solved as explained in iamwavecut#1