DiffEdit: Diffusion-based semantic image editing with mask guidance

[clarencechen]

Model/Pipeline/Scheduler description

DiffEdit is a zero-shot inpainting method given a text-to-image denoising diffusion model, requiring no additional fine-tuning or parameters.

The main innovations pioneered in DiffEdit are:

• Automated mask inference given the original and new conditioning text using the spatial distribution of the difference in conditioned noise estimates, highlighting the locations that are predicted to change the most between conditioning on the original and new texts.
• Using inverse sampling from model-estimated noise to provide a noised version of the unmasked portion of the input image for each sampling step, as opposed to mixing the unmasked portion with random IID Gaussian noise as in SDEdit and RePaint.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

• Link to DiffEdit paper: https://arxiv.org/pdf/2210.11427.pdf
• Notebook Implementation by @Xiang-cd based on Comp-vis: https://github.com/Xiang-cd/DiffEdit-stable-diffusion/blob/main/diffedit.ipynb. For better sample efficiency, they use DPM-solver as the deterministic sampler instead of DDIM.
• Model weights are not necessary for this pipeline as it is a zero-shot method given a text-to-image denoising diffusion model.

[sayakpaul]

Looks pretty cool!

Would you be open to opening a PR to add it?

Also, leaving a sample result from the paper for anyone interested in visually knowing what the method does:

[silvererudite]

pretty cool...I'm working on img2img applications and this looks super useful. Can I contribute here @sayakpaul ? What would be the steps to go about adding this pipeline/model pls?

[clarencechen]

@sayakpaul @silvererudite Thanks for the quick feedback! 🤗 I'm working on a PR to implement DiffEdit right now as part of a project, and I expect to be most of the way there by the end of next week.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.