ControlNet broken from_single_file

[vladmandic]

Describe the bug

controlnet loader from_single_file was originally added via #4084
and method ControlNet.from_single_file() works for non-converted controlnets.

but for controlnets in safetensors format that contain already converted state_dict, it errors out.

its not reasonable to expect from user to know what is the internal dict structure of the controlnet safetensors file
before he can use it.

even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format
which makes them impossible to load in difufsers.
for example: https://huggingface.co/Laxhar/noob_openpose/tree/main

this issue was already mentioned several times, each time closed as "works as designed"
when in reality its just a failure that should be addressed as an issue.
see #8474 #9208 #8614 as examples of previous issues

Reproduction

scenario-1: works with non-converted controlnet

import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='Aptronym/SDNext', filename='ControlNet11/controlnet11Models_canny.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)

scenario-1: fails for majority of controlnets available on huggingface

import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='lllyasviel/sd_control_collection', filename='diffusers_xl_canny_small.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)

initial failure is nonsense

OSError: stable-diffusion-v1-5/stable-diffusion-v1-5 does not appear to have a file named config.json.

whats making this worse is that SD15 and SDXL share the same ControlNet class which causes some
confusion on the base repo where to lookup config.
e.g,, here we're loading SDXL controlnet and error referrs to SD15 repo.

anyhow, trying to force correct config:

cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16, config='diffusers/controlnet-canny-sdxl-1.0-small')

results in even worse nonsense failure during loading of state_dict:

TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not NoneType

System Info

diffusers=0.32.0.dev0
python==3.12.3
torch==2.5.1+cu124

Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza

[sayakpaul]

even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format
which makes them impossible to load in difufsers.
for example: https://huggingface.co/Laxhar/noob_openpose/tree/main

Isn't this an actual error as it is partially in the diffusers format by not including the config.json?

[vladmandic]

@sayakpaul cannot blame users for not knowing internals of diffusers.
99% of users will attempt to download that safetensors file and load it - and it will fail.
sure, author of that specific repo should add config.json (and yes, request to author is already created),
but are we going to re-educate every user "don't download controlnet safetensors"?
(and there is no way to differentiate from user perspective which ones can and which cannot be downloaded)

[sayakpaul]

In that case what looks like a good solution to you? I will let @DN6 to of course comment here but IMO the following could be good candidates:

• Detect if this is a diffusers checkpoint and if the supplied path doesn't have a config.json. If detected, yield a nice error message.
• Detect if this is a diffusers checkpoint and try to fetch a config.json but I am not sure how well this would play out as it involves quite a bit of guess work from what I can see.

[vladmandic]

Detect if this is a diffusers checkpoint and try to fetch a config.json but I am not sure how well this would play out as it involves quite a bit of guess work from what I can see.

which guesswork? is it sd15 or sdxl should be easy to infer from state_dict itself.
that's how it works when loading main dffusion model using from_single_file. if i were a user, i would have same expectation here.

[sayakpaul]

Hmm could be. I thought too fast. So, fetch the config, load the state dict -- looks like the workflow here.

with init_empty_weights():
    config = ControlNetModel.load_config("...")
    controlnet_model = ControlNetModel.from_config(config)
    controlnet_model.load_state_dict(diffusers_state_dict)

[vladmandic]

Hmm could be. I thought too fast. So, fetch the config, load the state dict -- looks like the workflow here.

btw, as a starting point, we can pass config=... arg to from_single_file and add autodetect later.
but need to make from_single_file actually work :)

[hlky]

In the first case the model config is being detected as v1 because the checked keys for controlnet are for non-Diffusers type checkpoints.

diffusers/src/diffusers/loaders/single_file_utils.py

Line 457 in f6f7afa

def infer_diffusers_model_type(checkpoint):

diffusers/src/diffusers/loaders/single_file_utils.py

Line 65 in f6f7afa

"controlnet": "control_model.time_embed.0.weight",

infer_diffusers_model_type can also be extended to detect e.g. v1 vs xl ControlNet.

In the second case convert_controlnet_checkpoint expects non-Diffusers type, the state dict it returns appears to only contain controlnet_mid_block.weight and controlnet_mid_block.bias as None which in turn causes the error when the state dict is loaded to the model.

convert_controlnet_checkpoint can return the provided checkpoint when Diffusers keys are detected.

diffusers/src/diffusers/loaders/single_file_utils.py

Lines 1169 to 1170 in f6f7afa

	new_checkpoint["controlnet_mid_block.weight"] = controlnet_state_dict.get("middle_block_out.0.weight")
	new_checkpoint["controlnet_mid_block.bias"] = controlnet_state_dict.get("middle_block_out.0.bias")

diffusers/src/diffusers/loaders/single_file_utils.py

Line 1070 in f6f7afa

def convert_controlnet_checkpoint(

diffusers/src/diffusers/models/model_loading_utils.py

Line 170 in f6f7afa

def load_model_dict_into_meta(