[Community Pipeline] RePaint + Stable Diffusion

[anton-l]

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Combining RePaint with Stable Diffusion

The idea of this pipeline is to improve upon the legacy stable diffusion inpainting pipeline by making use of the RePaint approach

At the moment the RePaint pipeline only works for image diffusion, as opposed to latent diffusion (as in Stable Diffision). But making the RePaint scheduler work with latent diffusion would be a great way to improve unsupervised inpainting results!

Resources:

• Legacy unsupervised inpainting pipeline: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint_legacy.py
• RePaint pipeline that doesn't support latent diffusion: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/repaint/pipeline_repaint.py
• Original RePaint project page and paper: https://github.com/andreas128/RePaint/

[nathanielherman]

Ooh I actually implemented this today https://github.com/huggingface/diffusers/pull/1341/files!

[Randolph-zeng]

Hi @nathanielherman , I am also playing around with RePaint(Implemented under DDIM) and SD. Have you noticed any performance issue with DDIM when doing repaint similar to this #1602 ?
I really love your implementation, but when trying your impl, I found some small glitches and I have pointed it out in the PR. Also I believe the repaint pipeline performance is also unsatisfying both in my local impl version and your version. I strongly suspect it is related to the DDIM issue in the diffusers library I mentioned above.
@anton-l Can you kindly suggest someone to look into this or help to check what is going on here ? Thanks a lot !

[Randolph-zeng]

And a minimum code snippet to reproduce the repaint stuff is here, might need only 1~2 minutes to set up this experiment.
Note repaint_ddim_scheduler and sd_repaint_pipeline are the pipeline and scheduler files I copied from the PR. You only need to change the relative import to the correct diffusers lib import to make following code snippet to work ( which should only takes you at most 2 minutes to create such local files). A quick note is that there are two small bugs I mentioned above, please check my comments in the PR to fix those two lines to make the following code snippet to work

import PIL
import requests
from io import BytesIO
from schedulers.repaint_ddim_scheduler import RePaintScheduler
from pipelines.sd_repaint_pipeline import StableDiffusionRepaintPipeline

def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url).resize((512, 512))
mask_image = download_image(mask_url).resize((512, 512))

pipe = StableDiffusionRepaintPipeline.from_pretrained('/cognitive_comp/common_data/stable-diffusion-v1-5').to(7)
pipe.scheduler = RePaintScheduler.from_config(pipe.scheduler.config)
prompt = "a bright lamp, high resolution, highly detailed, placed on a park bench"
outpiant_res = pipe(init_image=init_image,
    mask_image=mask_image,
    prompt=prompt,
    num_inference_steps=100,
    guidance_scale=2.5,
    jump_length=1,
    jump_n_sample=2)
outpiant_res.images[0]

[sayakpaul]

Closing because of #3320