convert_lora_safetensor_to_diffusers

[NicholasKao1029]

this script https://github.com/huggingface/diffusers/blob/main/scripts/convert_lora_safetensor_to_diffusers.py integrates the lora into the Pipeline and then outputs said Pipeline.

Is there a method to turn the .safetensors Lora to a .bin file to be loaded in dynamically similar to the snippet below

from diffusers import StableDiffusionPipeline
import torch

model_path = "sayakpaul/sd-model-finetuned-lora-t4"
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16)
pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

prompt = "A pokemon with blue eyes."
image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5, cross_attention_kwargs={"scale": 0.5}).images[0]
image.save("pokemon.png")

As seen here https://huggingface.co/docs/diffusers/training/lora

My complaint is there is a lot of redundancy in the model if you want different loras to be used.

[sayakpaul]

Cc: @patrickvonplaten @Narsil

[Narsil]

The conversion of the LoRA should only be a key remapping right ?

load_attn_procs already supports loading LoRA from safetensors file. I think the problem is that SD-safetensors != diffusers-safetensors. (The name of the tensors are different). Am I correct ?

[NicholasKao1029]

I'm not sure this is the case, as the call to the convert function outputs a Pipe,

diffusers/scripts/convert_lora_safetensor_to_diffusers.py

Line 125 in 1d7b4b6

pipe = convert(base_model_path, checkpoint_path, lora_prefix_unet, lora_prefix_text_encoder, alpha)

This outputs a directory much like https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main which is a full model

whereas in the load_attn_procs example the use of https://huggingface.co/sayakpaul/sd-model-finetuned-lora-t4/tree/main has a different structure specifically it has pytorch_lora_weights.bin

[NicholasKao1029]

Sorry I misunderstood @Narsil, you are right it is only a key remapping,
However my point still holds true as the output is ultimately a full model

[sayakpaul]

IIUC we output the full model for generality across different pipelines and also to keep the output artifacts to be as self-contained as possible. If you have some ideas on how we can improve this bit, feel free to open a PR :)

@patrickvonplaten Cc.

[fungiboletus]

I was expecting the script to generate a lora and not a whole model. Is it possible to convert the safetensors lora to diffusers lora? Or should we wait for the next release that should support safetensors lora without conversion ?

Edit: I found a conversion script there: https://github.com/haofanwang/Lora-for-Diffusers/blob/18adfa4da0afec46679eb567d5a3690fd6a4ce9c/format_convert.py#L154-L161

[YulienPohl]

Edit: I found a conversion script there: https://github.com/haofanwang/Lora-for-Diffusers/blob/18adfa4da0afec46679eb567d5a3690fd6a4ce9c/format_convert.py#L154-L161

For me this script does not work, it returns a 48 MB bin from a 144 MB safetensor.
And when I use it, it only creates images with colored spots.

[patrickvonplaten]

We should try to have an improved load_lora function that also allows to load from the orig LoRA format (similar to how we do it here: #2009 (comment)).

[NicholasKao1029]

@patrickvonplaten this would be fantastic!

[patrickvonplaten]

Related PR: #2882

[NicholasKao1029]

Updated PR: #2918

[sayakpaul]

Regarding the safetensors LoRA thing, does it make sense?
#2866 (comment)

[NicholasKao1029]

Regarding the safetensors LoRA thing, does it make sense? #2866 (comment)

Sorry for the late reply @sayakpaul.

If I understand the linked comment, that would be for .bin -> safetensors for loras. Is there a corresponding safetensors -> .bin planned here or perhaps I misunderstood