[OmegaConf] replace it with yaml

[sayakpaul]

What does this PR do?

Since PyYAML needs to be installed anyway with huggingface_hub which is a required dependency for diffusers, it makes sense to replace the usage of omegaconf with yaml. We use omegaconf to deal with the config files of the single file checkpoints.

If there's agreement I will make the changes to the rest of the files and remove omegaconf entirely from diffusers.

[sayakpaul]

Hmm, needs a bit more work, will look into it.

If I do:

from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_single_file(
    "https://huggingface.co/WarriorMama777/OrangeMixs/blob/main/Models/AbyssOrangeMix/AbyssOrangeMix.safetensors"
)
pipeline.to("cuda")

_ = pipeline("hey", num_inference_steps=5).images[0]

It fails with:

Traceback (most recent call last):
  File "/home/sayak/check_without_omegaconf.py", line 4, in <module>
    pipeline = StableDiffusionPipeline.from_single_file(
  File "/home/sayak/.pyenv/versions/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/sayak/diffusers/src/diffusers/loaders/single_file.py", line 257, in from_single_file
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/home/sayak/diffusers/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1321, in download_from_original_stable_diffusion_ckpt
    and "cond_stage_config" in original_config.model.params
AttributeError: 'dict' object has no attribute 'model'

[sayakpaul]

Okay, so, to get around the AttributeError: 'dict' object has no attribute 'model' problem, I did:

class DotDict(dict):
    """ Custom dictionary class to access dictionary elements using dot notation. """
    __getattr__ = dict.get
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

    def __init__(self, dictionary):
        for key, value in dictionary.items():
            if isinstance(value, dict):
                value = DotDict(value)
            self[key] = value

So, it makes member access with "." possible. For example:

from io import BytesIO
import yaml 
import requests 

config_url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/configs/stable-diffusion/v1-inference.yaml"

original_config_file = BytesIO(requests.get(config_url).content)
original_config  = yaml.safe_load(original_config_file)
original_config = DotDict(original_config)

Now, we can do:

original_config.model.params

You can also do original_config["model"]["params"], of course.

So, I think we have two options:

• Use PyYAML like the above.
• Change calls like original_config.model.params to original_config["model"]["params"]. This requires more code changes.

[patrickvonplaten]

I'm not sure what exactly to review here - does PyYaml already work?

[sayakpaul]

Let me know which option in #6488 (comment) you'd prefer in light of #6488 (comment)?

[patrickvonplaten]

I think it's totally fine if we do original_config["model"]["params"]! Nice to see that huggingface_hub uses pyyaml already anyways!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

@patrickvonplaten I have replaced OmegaConf completely from the scripts concerning Stable Diffusion and ControlNet. Have gone ahead and also tested whatever the following caught up:

If the changes are okay, I will propagate them to the other scripts and utilities. LMK.

[patrickvonplaten]

Changes look great! Nice job

[DN6]

Amazing work! 👍🏽

[patrickvonplaten]

Great job!

[vladmandic]

this seems to cause pretty bad regressions in some areas and its preventing model load.
for example:

│ /home/vlado/.local/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py:1326 in download_from_original_stable_diffusion_ckpt                                                                                                                                                                                   │
│                                                                                                                                                                                                                                                                                                                                                  │
│   1325 │   │   model_type is None                                                                                                                                                                                                                                                                                                                │
│ ❱ 1326 │   │   and "cond_stage_config" in original_config["model"]["params"]                                                                                                                                                                                                                                                                     │
│   1327 │   │   and original_config["model"]["params"]["cond_stage_config"] is not None

TypeError: string indices must be integers, not 'str'

[sayakpaul]

I did run these tests from #6488 (comment). Could you please give me something that helps to reproduce the problem? I will fix right away :)

[vladmandic]

simple loading using from_single_file fails with diffusers==0.26.0.dev and works with diffusers==0.25.1

import time
import torch
import diffusers

model_file = '/mnt/d/Models/stable-diffusion/base/sd-v15-runwayml.safetensors'
load_config = {
    "low_cpu_mem_usage": True,
    "torch_dtype": torch.float16,
    "safety_checker": None,
    "requires_safety_checker": False,
    "load_safety_checker": False,
    "load_connected_pipeline": True,
    'config_files': {
        'v1': 'configs/v1-inference.yaml',
        'v2': 'configs/v2-inference-768-v.yaml',
        'xl': 'configs/sd_xl_base.yaml',
        'xl_refiner': 'configs/sd_xl_refiner.yaml',
    }
}

t0 = time.time()
pipe = diffusers.StableDiffusionPipeline.from_single_file(model_file, **load_config)
t1 = time.time()
print(f"Loaded: {t1-t0:.2f}")

note: i specify config files in load config to prevent load perform a network lookup/download on each model load.

[sayakpaul]

Ah. Tracking in #6615.

Cc: @DN6 might be worth adding a test with a local config file.

[sayakpaul]

It works with #6633, thanks to @spezialspezial!

Colab: https://colab.research.google.com/gist/sayakpaul/99e1759ca450fa39b47b647f1e2fbd6f/scratchpad.ipynb

[jasstionzyf]

@sayakpaul
module= AutoencoderKL.from_single_file(modelPath_,config_file=config_file,torch_dtype=torch.float16)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/apps/miniconda3/envs/sd/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/data/projects/diffusers/src/diffusers/loaders/single_file.py", line 449, in from_single_file
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/projects/diffusers/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 348, in create_vae_diffusers_config
vae_params = original_config["model"]["params"]["first_stage_config"]["params"]["ddconfig"]
~~~~~~~~~~~~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'`

[sayakpaul]

@jasstionzyf can you ensure you are using the latest main? #6633 should have already fixed it.

[jasstionzyf]

@sayakpaul i use pip install git+https://github.com/huggingface/diffusers try to build and install latest main branch , but result is alwayes 0.26.0.dev0, is that normal?

[sayakpaul]

Ah I see.

You should try the following:

pip uninstall diffusers -y
git clone https://github.com/huggingface/diffusers
cd diffusers && pip install -e .

[jasstionzyf]

@sayakpaul
Still above errors.
And also i have checked my error is not same as this fix's errors

[sayakpaul]

Then please open a new issue and provide a fully reproducible yet minimal script. Please keep it as minimal as possible while also making it fully reproducible.