StableDiffusion3Pipeline `load_lora_weights` IndexError

[Beinsezii]

Describe the bug

Using load_lora_weights with a single safetensor on StableDiffusion3Pipeline results in an IndexError

Reproduction

StableDiffusion3Pipeline fails at load_lora_weights with IndexError on latest diffusers master

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained('stabilityai/stable-diffusion-3-medium-diffusers', torch_dtype=torch.float16).to('cuda')

pipe("photo of a woman in a red dress", generator=torch.manual_seed(1)).images[0].save('no_lora.png')

pipe.load_lora_weights("pytorch_lora_weights.safetensors", adapter_name="test_lora")
pipe.set_adapters(["test_lora"], [1.0]) # Also not implemented yet. Being worked on.
pipe.fuse_lora(["test_lora"])

pipe("photo of a woman in a red dress", generator=torch.manual_seed(1)).images[0].save('yes_lora.png')

Same script succeeds on other pipelines

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained('stabilityai/stable-diffusion-xl-base-1.0', torch_dtype=torch.float16).to('cuda')

pipe("photo of a woman in a red dress", generator=torch.manual_seed(1)).images[0].save('no_lora.png')

pipe.load_lora_weights("pytorch_lora_weights-xl.safetensors", adapter_name="test_lora")
pipe.set_adapters(["test_lora"], [1.0])
pipe.fuse_lora(["test_lora"])

pipe("photo of a woman in a red dress", generator=torch.manual_seed(1)).images[0].save('yes_lora.png')

Logs

Traceback (most recent call last):
  File "/home/beinsezii/Python/quickdif/repro.py", line 8, in <module>
    pipe.load_lora_weights("pytorch_lora_weights.safetensors", adapter_name="test_lora")
  File "/home/beinsezii/Python/quickdif/diffusers/loaders/lora.py", line 1387, in load_lora_weights
    self.load_lora_into_transformer(
  File "/home/beinsezii/Python/quickdif/diffusers/loaders/lora.py", line 1555, in load_lora_into_transformer
    lora_config_kwargs = get_peft_kwargs(rank, network_alpha_dict=None, peft_state_dict=state_dict)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beinsezii/Python/quickdif/diffusers/utils/peft_utils.py", line 153, in get_peft_kwargs
    r = lora_alpha = list(rank_dict.values())[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

System Info

• 🤗 Diffusers version: 0.30.0.dev0
• Platform: Linux-6.9.4-arch1-1-x86_64-with-glibc2.39
• Running on a notebook?: No
• Running on Google Colab?: No
• Python version: 3.12.3
• PyTorch version (GPU?): 2.3.1+rocm6.0 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.23.3
• Transformers version: 4.41.2
• Accelerate version: 0.31.0
• PEFT version: 0.11.1
• Bitsandbytes version: 0.43.2.dev
• Safetensors version: 0.4.3
• xFormers version: not installed
• Accelerator: NA
• Using GPU in script?: gfx1100
• Using distributed or parallel set-up in script?: False

Who can help?

@DN6

[asomoza]

The SD3 Pipeline inherits SD3LoraLoaderMixin which doesn't have a set_adapters so actually the error should be:

AttributeError: 'StableDiffusion3Pipeline' object has no attribute 'set_adapters'

The error you're getting is not with the set_adapters but with the load_lora_weights in this line:

lora_config_kwargs = get_peft_kwargs(rank, network_alpha_dict=None, peft_state_dict=state_dict)

Did you train the Lora with the diffusers training code? I trained one and I can load it without problems, so maybe there was something wrong with the training.

[bghira]

i think he's testing with ptx0/sd3-lora-test which was trained with a port of the diffusers code to simpletuner, but it works with this code and no errors:

from diffusers import StableDiffusion3Pipeline
import torch

pipe = StableDiffusion3Pipeline.from_pretrained('ptx0/sd3-reality-mix').to(device='mps', dtype=torch.float16)
pipe.load_lora_weights('ptx0/sd3-lora-test')
image = pipe(
    prompt="a lion singing in the shower",
    num_inference_steps=30,
    guidance_scale=5.5, width=768, height=768
).images[0]
image.save('lion.png', format='PNG')

[Beinsezii]

Updated title and body to correct function.

Appears as though the target issue will likely be resolved in #8584

While set_adapters is pending a larger refactor.

[sayakpaul]

Thanks @Beinsezii! Yes, set_adapters() will land very soon :) Stay tuned.

[Beinsezii]

Original issue confirmed resolved with e7b9a07

The repro code now runs with the set_adapters() call disabled