ControlNet from_single_file when already converted

[hlky]

What does this PR do?

This PR fixes #9976

ControlNet from_single_file is broken when used with a checkpoint that has already been converted:

import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='lllyasviel/sd_control_collection', filename='diffusers_xl_canny_small.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)

OSError: stable-diffusion-v1-5/stable-diffusion-v1-5 does not appear to have a file named config.json.

diffusers/src/diffusers/loaders/single_file_utils.py

Line 457 in f6f7afa

def infer_diffusers_model_type(checkpoint):

diffusers/src/diffusers/loaders/single_file_utils.py

Line 65 in f6f7afa

"controlnet": "control_model.time_embed.0.weight",

The state dict is incorrectly being detected as v1 Stable Diffusion because the checked keys for controlnet are for non-Diffusers format.

We add to checked keys for controlnet and add controlnet_xl_large, controlnet_xl_mid and controlnet_xl_small as the final types with controlnet_xl being used internally as a key to check for XL vs v1.

controlnet_xl keys are for Diffusers type checkpoints and need equivalents for non-converted checkpoints.

When fixed, or if we provide the correct config:

cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16, config='diffusers/controlnet-canny-sdxl-1.0-small')

TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not NoneType

convert_controlnet_checkpoint called as part of from_single_file expects non-Diffusers type, the conversion function runs without error and returns an unusable state dict.

convert_controlnet_checkpoint is modified to return the checkpoint if it's already been converted.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul @vladmandic

[vladmandic]

nice! did a quick test and seems fine.

[yiyixuxu]

thanks @hlky !
can you take a look here? @DN6

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Thanks @hlky. Would it be possible to do a follow up for the ControlNetXL small version?

[hlky]

Small, mid and large/full versions of ControlNet XL should all be covered, I've tested a couple checkpoints for each. I haven't found any ControlNet XL checkpoints that aren't already in Diffusers format but I'll add non-Diffusers keys to check in a follow up, and I'll check for something specific to detect small version - currently small is selected if known keys in medium and large versions aren't detected.

import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download

small = 'diffusers/controlnet-canny-sdxl-1.0-small'
mid = 'diffusers/controlnet-canny-sdxl-1.0-mid'
large = 'diffusers/controlnet-canny-sdxl-1.0'

for size in {small, mid, large}:
    local_path = hf_hub_download(repo_id=size, filename='diffusion_pytorch_model.fp16.safetensors')
    cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
    print(cn.__class__)

small = 'diffusers_xl_canny_small.safetensors'
mid = 'diffusers_xl_canny_mid.safetensors'
large = 'diffusers_xl_canny_full.safetensors'

for size in {small, mid, large}:
    local_path = hf_hub_download(repo_id="lllyasviel/sd_control_collection", filename=size)
    cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
    print(cn.__class__)

# xl large/full
local_path = hf_hub_download(repo_id="Laxhar/noob_openpose", filename="openpose_pre.safetensors")
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)

# v1
local_path = hf_hub_download(repo_id="lllyasviel/sd-controlnet-canny", filename="diffusion_pytorch_model.safetensors")
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)