Improve support of Reference-only and outpainting for SDXL

[patrickvonplaten]

Reference-only has shown be a very powerful mechanism for outpainting as well as image variation. We should investigate a bit how we can best support this in a modularized, library-friendly way in diffusers. Also I think we should try this out for SDXL.

Related links:

• [New Preprocessor] The "reference_adain" and "reference_adain+attn" are added Mikubill/sd-webui-controlnet#1280
• [1.1.202 Inpaint] Improvement: Everything Related to Adobe Firefly Generative Fill Mikubill/sd-webui-controlnet#1464
• [Major Update] Reference-only Control Mikubill/sd-webui-controlnet#1236

and

https://github.com/huggingface/diffusers/blob/main/examples/community/stable_diffusion_controlnet_reference.py
https://github.com/huggingface/diffusers/blob/main/examples/community/stable_diffusion_reference.py

@DN6 maybe this could be a first fun project for you

[bghira]

SDXL 1.5 or SD 1.5? I got very excited for a moment, but now I'm not sure 😅

[patrickvonplaten]

SD XL sorry 😅

[yamkz]

I have always wanted this feature. I wish it could be included in the diffuser library, with support for SDXL.
#3872

[yanchaoguo]

+1

[DN6]

I think this PR: #4502 is relevant. SDXL reference pipeline will be added to Community Pipelines

[sayakpaul]

@DN6 could you maybe jam on #4502 then to help get it merged? But I concur with Patrick that we should experiment with this ourselves and make it a part of the core library.

[VigneshHexo]

Reference-only has shown be a very powerful mechanism for outpainting as well as image variation

This one @sayakpaul

[DN6]

SDXL Reference is now available as a Community Contributed Pipeline in #4502. Opened a separate ticket for outpainting: #4718.

Closing this issue for now.

[patrickvonplaten]

I think we should dive a bit deeper here and run some experiments. It would be really nice to have a fully working outpainting workflow for SDXL

[patrickvonplaten]

Also cc @williamberman maybe inpainting can help here as well

[VigneshHexo]

I think we should dive a bit deeper here and run some experiments. It would be really nice to have a fully working outpainting workflow for SDXL

Will be very happy to help with this! If there is a specific to do list, will pick it up from there and get it done! Otherwise if someone is working on this, would be very happy to tag along and help them out! Please let me know @patrickvonplaten