Add Differential Diffusion to Kolors

[saqlain2204]

What does this PR do?

Adds Differential Diffusion to Kolors

Fixes Partially #8924

@a-r-r-o-w @DN6

import torch

from diffusers.utils import load_image
from pipeline_kolors_differential_img2img import KolorsDifferentialImg2ImgPipeline

pipe = KolorsDifferentialImg2ImgPipeline.from_pretrained(
             "Kwai-Kolors/Kolors-diffusers", variant="fp16", torch_dtype=torch.float16
       )
pipe.enable_model_cpu_offload()

source_image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png"
)
map = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask_2.png"
)
prompt = "a green pear"
negative_prompt = "blurry"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=source_image,
    num_inference_steps=28,
    guidance_scale=4.5,
    strength=1.0,
    map=map,
).images[0]

[saqlain2204]

I am not able to load kolors in colab free tier. Tried to resolve it by using memory optimizations, but it dint work. Am I going wrong somewhere?

[a-r-r-o-w]

Thanks for working on this! For Kolors, I think @asomoza might be able to share some suggestions on how to save memory and minimal VRAM requirements as he worked on it. If it doesn't work on a free-tier colab, I'll try and test the PR :)

[saqlain2204]

Sure. Thanks :)

[asomoza]

Hi @saqlain2204 sorry I missed this for so long. Kolors uses a big text encoder and a custom one that isn't in transformers yet so we can't optimize it for now, so for the time being, we can't use a quantized version of it.

As I remember the text encoder is around 10GB and the unet 6 GB so they won't fit in the free tier colab GPU, don't remember how much RAM it has to store them in the CPU, but it should work with enable_model_cpu_offload. There was a problem before where the VRAM wasn't getting freed but that was fixed a while ago, maybe you can try again.

If it doesn't work like this, sadly we won't be able to optimize it just yet so it can run on the free tier colab.

[yiyixuxu]

should we merge this? @asomoza

[saqlain2204]

Hi @asomoza ,
Thanks for the suggestions. Will try running it by making the above changes and update it here.

[asomoza]

@saqlain2204 can you update the description with the typical examples we use for diff-diff, you can get them from the SD3 implementation.

This way you and we can make sure it works which is pretty much the only requirement for community pipelines. From a quick look, it seems you're missing the map/mask input so there's no way to pass the depthmap/mask to the pipeline call method.

[saqlain2204]

I have updated. Can you please review it.

[asomoza]

we need a couple of fixes, the rest looks good

[asomoza]

@saqlain2204 can you please run make style and make quality

[asomoza]

thanks!

[saqlain2204]

👍

[saqlain2204]

Thanks @a-r-r-o-w @asomoza !