Skip to content

Stabilise LMS and further stabilise DPM++ to eliminate residual noise in the result #5690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from

Conversation

nhnt11
Copy link

@nhnt11 nhnt11 commented Nov 7, 2023

What does this PR do?

This PR makes two changes:

  1. For the LMS scheduler, introduces a parameter that forces order=1 for the last 15 steps. This makes LMS usable with Image to Image with strength closer to 0 (e.g. 0.1). This solves LMS scheduler leaves a lot of noise leftover in the result image when used with SDXL Img2Img Pipeline #5630 (more example results in that issue's thread)
  2. For the DPMSolverMultistep scheduler, we force the last sigma to be 0.0 if euler_at_final is true. This prevents residual noise in the result image. This solves DPM++ leaves residual noise in SDXL images, which is unexpected #5689 (more example results in that issue's thread)

Fixes #5630, #5689

@nhnt11
Copy link
Author

nhnt11 commented Nov 7, 2023

I am still working on cleaning this up a bit and updating documentation for the LMS config param 🙏

@patrickvonplaten
Copy link
Contributor

@yiyixuxu could you take a look here?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@patrickvonplaten
Copy link
Contributor

Gentle ping here @yiyixuxu

@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Jan 7, 2024

hi sorry I didn't have time to look into this sooner. We got a couple more bug reports on DPM scheduler with use_karras_sigmas, so I looked into it more. I think there was an issue in our implementation with use_karras_sigmas, and you are absolutely correct that the last sigma should be 0 (that's how it is done in k-diffusions)

I'm working on fixing it with PR here #6477

@nhnt11 nhnt11 closed this Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LMS scheduler leaves a lot of noise leftover in the result image when used with SDXL Img2Img Pipeline
5 participants