[SD-XL] Add inpainting

[patrickvonplaten]

SD-XL inpainting

This PR solves: #4080 and is ready for a review.

Inpainting works well for both the vanilla case and the "Ensemble of Expert Denoisers case".

You can try the following to see for yourself:

Vanilla inpainting:

import torch
from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipe.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A red cat sitting on a bench"
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80).images[0]

Ensemble of Expert of denoisers

which should give slightly better quality:

from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipe.to("cuda")

refiner = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-0.9",
    text_encoder_2=pipe.text_encoder_2,
    vae=pipe.vae,
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)
refiner.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A red cat sitting on a bench"
num_inference_steps = 75
high_noise_frac = 0.7

image = pipe(
    prompt=prompt,
    image=init_image,
    mask_image=mask_image,
    num_inference_steps=num_inference_steps,
    strength=0.80,
    denoising_start=high_noise_frac,
    output_type="latent",
).images
image = refiner(
    prompt=prompt,
    image=image,
    mask_image=mask_image,
    num_inference_steps=num_inference_steps,
    denoising_start=high_noise_frac,
).images[0]

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[gkorepanov]

Hi, is there already a SD XL checkpoint with unet having 9 channels? Seems like no specific inpainting model was released for SD XL, but without it inpainting results are meaningless (I mean there is little to no semantic match between inpainted regions and present regions in generated images)

[AmericanPresidentJimmyCarter]

Is this the same method as InpaintLegacy in SD? There are now 3 inpainting methods for LDM, the "InpaintLegacy" method, the model using extra channels from RunwayML, and PSLD. I think we should maintain consistent naming.

[AmericanPresidentJimmyCarter]

And yes, I agree with @gkorepanov , the "InpaintLegacy" method is more or less useless.

[patrickvonplaten]

StableDiffusionInpaintPipelineLegacy is deprecated and will be removed. Everything you were able to do with StableDiffusionInpaintPipelineLegacy you can now do with StableDiffusionInpaintPipeline

In that sense there will only be one "true" StableDiffusionXLInpaintPipeline

[adhikjoshi]

As inpaiting checkpoint isn't there, does it affects quality in general?

[patrickvonplaten]

As inpaiting checkpoint isn't there, does it affects quality in general?

Works pretty well for me for now, I recommend making sure to pass strength=0.7 or strength=0.8.

I think the checkpoint will however have problems when you want to replace the masked area with something very different to what was there before.

[nhnt11]

Is it really intended to use the refiner model for general img2img? I've been trying to understand this - I've also seen it here for example - but I think I am missing something. My understanding is that the refiner model is intended as a kind of de-noising and/or fidelity-increasing step and it isn't good at generating the kind of baseline content of the image. If that's correct, feels like it'd perform poorly for img2img with lower strength values.

[patrickvonplaten]

You can use both! The refiner might be better suited for images that look already like the prompt which is the case here. We should maybe improve the docs after the official release.

[nhnt11]

Gotcha - thanks for clarifying!

[patrickvonplaten]

This is never used actually and was a copy-paste bug I think