Cannot load most controlnet models using from_single_file()

[lstein]

Describe the bug

The controlnet models found at https://huggingface.co/lllyasviel/sd_control_collection fail to load with from_single_file(). The error message is OSError: runwayml/stable-diffusion-v1-5 does not appear to have a file named config.json.

Maybe these models are not intended to be loaded with from_single_file ?

Reproduction

Here is an example derived from the ControlNetModel reference docs:

from diffusers import ControlNetModel

working_url = "https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_canny.pth"
controlnet = ControlNetModel.from_single_file(working_url)
print("success 1")

not_working_url = "https://huggingface.co/lllyasviel/sd_control_collection/blob/main/kohya_controllllite_xl_blur.safetensors"
controlnet = ControlNetModel.from_single_file(not_working_url)
print("success 2")

Logs

% python notes/controlnet_bug.py
success 1
Traceback (most recent call last):
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 379, in load_config
    config_file = hf_hub_download(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1282, in _hf_hub_download_to_cache_dir
    (url_to_download, etag, commit_hash, expected_size, head_call_error) = _get_metadata_or_catch_error(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata
    r = _request_wrapper(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper
    response = _request_wrapper(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 396, in _request_wrapper
    hf_raise_for_status(response)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 315, in hf_raise_for_status
    raise EntryNotFoundError(message, response) from e
huggingface_hub.utils._errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-66c0eb8e-2da273d634c5f89627857c32;788325fa-b84e-4291-b366-0ea930d21d23)

Entry Not Found for url: https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/config.json.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/notes/controlnet_bug.py", line 8, in <module>
    controlnet = ControlNetModel.from_single_file(not_working_url)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/diffusers/loaders/single_file_model.py", line 268, in from_single_file
    diffusers_model_config = cls.load_config(
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 406, in load_config
    raise EnvironmentError(
OSError: runwayml/stable-diffusion-v1-5 does not appear to have a file named config.json.

System Info

- 🤗 Diffusers version: 0.30.0
- Platform: Linux-5.15.0-113-generic-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.23.5
- Transformers version: 4.41.1
- Accelerate version: 0.31.0
- PEFT version: 0.11.1
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: 0.0.25.post1
- Accelerator: NVIDIA GeForce RTX 4070, 12282 MiB
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Who can help?

@sayakpaul @yiyixuxu @DN6

[tolgacangoz]

AFAIK, ControlNet-LLLite models aren't officially supported in diffusers.

[lstein]

The same issue occurs with SargeZT/controlnet-sd-xl-1.0-depth-16bit-zoe, which can be loaded with from_pretrained(), and has a corresponding single file. I think single file loading of this controlnet should be supported?

from diffusers import ControlNetModel

not_working_url = "https://huggingface.co/SargeZT/controlnet-sd-xl-1.0-depth-16bit-zoe/blob/main/depth-zoe-xl-v1.0-controlnet.safetensors"
controlnet = ControlNetModel.from_single_file(not_working_url)

It returns the runwayml/stable-diffusion-v1-5 does not appear to have a file named config.json error.

I also tried forcing config to use the diffuser repo's config.json:

from diffusers import ControlNetModel

not_working_url = "https://huggingface.co/SargeZT/controlnet-sd-xl-1.0-depth-16bit-zoe/blob/main/depth-zoe-xl-v1.0-controlnet.safetensors"
controlnet = ControlNetModel.from_single_file(not_working_url,
                                              config='SargeZT/controlnet-sd-xl-1.0-depth-16bit-zoe',
                                              )
print("success 1")

This produced a different error:

Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/notes/controlnet_bug.py", line 4, in <module>
    controlnet = ControlNetModel.from_single_file(not_working_url,
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/diffusers/loaders/single_file_model.py", line 299, in from_single_file
    unexpected_keys = load_model_dict_into_meta(model, diffusers_format_checkpoint, dtype=torch_dtype)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/diffusers/models/model_loading_utils.py", line 152, in load_model_dict_into_meta
    if empty_state_dict[param_name].shape != param.shape:
AttributeError: 'NoneType' object has no attribute 'shape'

[asomoza]

when you say:

and has a corresponding single file

You're just linking to the safetensors file inside the same repo which is a diffusers controlnet. This is not how it works, from_single_file is referring as to load a original format controlnet not a diffusers one without a config.json.

If you're referring to the one in the collections repo, it's the same diffusers one saved with fp16, you can look at the keys, so it's a diffusers controlnet that was stripped of the config and put in the same repo as the others. Most of the UIs often do this because the hardcode the config files to simplify the "distribution" but mostly no one bothers to do a real conversion to the original format.

The only ones that aren't supported in diffusers are the lllite and lora ones.

One solution you can do is to convert that controlnet to a real original format and then you can use from_single_file with it.

[lstein]

Thanks. The second case ("the one in the collections repo") seems to be causing the issue.