[2064]: Add Karras to DPMSolverMultistepScheduler #3001

nipunjindal · 2023-04-06T17:50:33Z

Discussion: #2064

Adding karras sigmas schedule to DPMSolverMultistepScheduler based on PR.

As I mentioned earlier in #2064, the sample_dpmp_sde is not equivalent to DPM Multistep + karras which is something I confirmed after running the test. There is another bug in the kdiffusion code where BrownianTreeNoiseSampler is not being passed the seed, so the run for sample_dpmp_sde gives back different results each time, fixed after giving the seed though.

I believe we can still merge this PR as it's needed but the #2064 needs a new scheduler.
Let me know your thoughts.

DPM Multistep + Karras Sigma

import diffusers
import torch
from diffusers import DPMSolverMultistepScheduler

seed = 33

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True
)

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]

display(image)

import torch
from diffusers import StableDiffusionKDiffusionPipeline

seed = 33

pipe = StableDiffusionKDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4"
).to("cuda")
pipe.set_scheduler("sample_dpmpp_2m")

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt, generator=generator, num_inference_steps=20, use_karras_sigmas=True
).images[0]

display(image)

HuggingFaceDocBuilderDev · 2023-04-06T17:57:06Z

The documentation is not available anymore as the PR was closed or merged.

nipunjindal · 2023-04-06T18:04:11Z

@patrickvonplaten @sayakpaul Would love to get your review and thoughts related to #2064. 🙏

sayakpaul

Hey @nipunjindal! Thanks a lot for adding in this one.

Maybe, let's also add a test?

sayakpaul · 2023-04-07T13:23:28Z

I believe we can still merge this PR as it's needed but the #2064 needs a new scheduler.

Also, could you elaborate the second part of ^ a bit?

nipunjindal · 2023-04-07T14:51:40Z

I believe we can still merge this PR as it's needed but the #2064 needs a new scheduler.

Also, could you elaborate the second part of ^ a bit?

Certainly. In response to this message, I believe that we need to introduce a new scheduler in diffusers that currently does not exist. I have previously mentioned this in my earlier message. I would like to confirm that you are in agreement with this proposal.

src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

patrickvonplaten

Looks good to me - let's maybe just make sure that the copy-from mechanism works correctly

Co-authored-by: Patrick von Platen <[email protected]>

sayakpaul · 2023-04-12T09:12:27Z

src/diffusers/schedulers/scheduling_deis_multistep.py

@@ -171,7 +171,6 @@ def __init__(
        self.model_outputs = [None] * solver_order
        self.lower_order_nums = 0

-    # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_timesteps


Because it's no longer the same (as we added use_karras_sigmas). So, in a separate PR, we can add support for Karras sigmans to this scheduler too.

Cc: @patrickvonplaten

Makes sense!

patrickvonplaten

Good to go for me once the tests are green :-)

patrickvonplaten · 2023-04-12T09:13:04Z

src/diffusers/schedulers/scheduling_deis_multistep.py

@@ -171,7 +171,6 @@ def __init__(
        self.model_outputs = [None] * solver_order
        self.lower_order_nums = 0

-    # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_timesteps


Makes sense!

nipunjindal · 2023-04-12T09:19:40Z

Sorry for being late to add the copied comment change. Thanks @sayakpaul for stepping in and helping. 🙏

sayakpaul · 2023-04-12T09:24:31Z

Sorry for being late to add the copied comment change. Thanks @sayakpaul for stepping in and helping. 🙏

It's no problem really. I was going to ping you. I hope it's fine I went into the PR a bit to make it fully ready to 🚀 We'd love to include this in the release we're planning.

sayakpaul · 2023-04-12T09:40:45Z

@nipunjindal the test is failing:

FAILED tests/schedulers/test_scheduler_dpm_multi.py::DPMSolverMultistepSchedulerTest::test_full_loop_with_karras_and_v_prediction - TypeError: 'NoneType' object cannot be interpreted as an integer

Possible for you to take a look?

nipunjindal · 2023-04-12T09:41:54Z

@sayakpaul Yes, I'll take a look and revert back.

nipunjindal · 2023-04-12T10:16:12Z

There seems to be an issue on the GitHub side as I'm unable to push or pull. Running ssh -vvvv, there are no errors shown, and I have attempted to resolve the problem by changing the SSH key to no avail. Here is the necessary modification that needs to be made due to the recent code change for num_inference_steps == num_timesteps, which led to inference_steps not being defined. While I investigate the issue with the GitHub connection, we can apply the patch below.

commit a8e05e84b8da0d61b55e00e3b73903c1b495f181
Author: njindal <[email protected]>
Date:   Wed Apr 12 15:25:33 2023 +0530

    Test Fix

diff --git a/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py b/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
index ef0aed55..3399ee2c 100644
--- a/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
+++ b/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
@@ -207,7 +207,7 @@ class DPMSolverMultistepScheduler(SchedulerMixin, ConfigMixin):
         if self.use_karras_sigmas:
             sigmas = np.array(((1 - self.alphas_cumprod) / self.alphas_cumprod) ** 0.5)
             log_sigmas = np.log(sigmas)
-            sigmas = self._convert_to_karras(in_sigmas=sigmas)
+            sigmas = self._convert_to_karras(in_sigmas=sigmas, num_inference_steps=num_inference_steps)
             timesteps = np.array([self._sigma_to_t(sigma, log_sigmas) for sigma in sigmas]).round()
             timesteps = np.flip(timesteps).copy().astype(np.int64)
 
@@ -285,14 +285,14 @@ class DPMSolverMultistepScheduler(SchedulerMixin, ConfigMixin):
         return t
 
     # Copied from diffusers.schedulers.scheduling_euler_discrete.EulerDiscreteScheduler._convert_to_karras
-    def _convert_to_karras(self, in_sigmas: torch.FloatTensor) -> torch.FloatTensor:
+    def _convert_to_karras(self, in_sigmas: torch.FloatTensor, num_inference_steps) -> torch.FloatTensor:
         """Constructs the noise schedule of Karras et al. (2022)."""
 
         sigma_min: float = in_sigmas[-1].item()
         sigma_max: float = in_sigmas[0].item()
 
         rho = 7.0  # 7.0 is the value used in the paper
-        ramp = np.linspace(0, 1, self.num_inference_steps)
+        ramp = np.linspace(0, 1, num_inference_steps)
         min_inv_rho = sigma_min ** (1 / rho)
         max_inv_rho = sigma_max ** (1 / rho)
         sigmas = (max_inv_rho + ramp * (min_inv_rho - max_inv_rho)) ** rho
diff --git a/src/diffusers/schedulers/scheduling_euler_discrete.py b/src/diffusers/schedulers/scheduling_euler_discrete.py
index eea1d14e..7237128c 100644
--- a/src/diffusers/schedulers/scheduling_euler_discrete.py
+++ b/src/diffusers/schedulers/scheduling_euler_discrete.py
@@ -206,7 +206,7 @@ class EulerDiscreteScheduler(SchedulerMixin, ConfigMixin):
             )
 
         if self.use_karras_sigmas:
-            sigmas = self._convert_to_karras(in_sigmas=sigmas)
+            sigmas = self._convert_to_karras(in_sigmas=sigmas, num_inference_steps=self.num_inference_steps)
             timesteps = np.array([self._sigma_to_t(sigma, log_sigmas) for sigma in sigmas])
 
         sigmas = np.concatenate([sigmas, [0.0]]).astype(np.float32)
@@ -241,14 +241,14 @@ class EulerDiscreteScheduler(SchedulerMixin, ConfigMixin):
         return t
 
     # Copied from https://github.com/crowsonkb/k-diffusion/blob/686dbad0f39640ea25c8a8c6a6e56bb40eacefa2/k_diffusion/sampling.py#L17
-    def _convert_to_karras(self, in_sigmas: torch.FloatTensor) -> torch.FloatTensor:
+    def _convert_to_karras(self, in_sigmas: torch.FloatTensor, num_inference_steps) -> torch.FloatTensor:
         """Constructs the noise schedule of Karras et al. (2022)."""
 
         sigma_min: float = in_sigmas[-1].item()
         sigma_max: float = in_sigmas[0].item()
 
         rho = 7.0  # 7.0 is the value used in the paper
-        ramp = np.linspace(0, 1, self.num_inference_steps)
+        ramp = np.linspace(0, 1, num_inference_steps)
         min_inv_rho = sigma_min ** (1 / rho)
         max_inv_rho = sigma_max ** (1 / rho)
         sigmas = (max_inv_rho + ramp * (min_inv_rho - max_inv_rho)) ** rho

Co-authored-by: njindal <[email protected]>

sayakpaul · 2023-04-12T10:38:02Z

Sorry, I think I probably messed up with the co-authorship. Apologies.

nipunjindal · 2023-04-12T11:52:24Z

These look unrelated error:

FAILED tests/pipelines/stable_diffusion/test_stable_diffusion.py::StableDiffusionPipelineFastTests::test_stable_diffusion_height_width_opt - Failed: Timeout >60.0s
FAILED tests/pipelines/stable_unclip/test_stable_unclip_img2img.py::StableUnCLIPImg2ImgPipelineFastTests::test_inference_batch_consistent - Failed: Timeout >60.0s
==== 2 failed, 1020 passed, 723 skipped, 391 warnings in 783.04s (0:13:03) =====

sayakpaul · 2023-04-12T12:34:38Z

I think that is unrelated and okay to merge :)

* [2737]: Add Karras DPMSolverMultistepScheduler * [2737]: Add Karras DPMSolverMultistepScheduler * Add test * Apply suggestions from code review Co-authored-by: Patrick von Platen <[email protected]> * fix: repo consistency. * remove Copied from statement from the set_timestep method. * fix: test * Empty commit. Co-authored-by: njindal <[email protected]> --------- Co-authored-by: njindal <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Sayak Paul <[email protected]>

njindal added 2 commits April 6, 2023 22:04

[2737]: Add Karras DPMSolverMultistepScheduler

eb696b9

[2737]: Add Karras DPMSolverMultistepScheduler

c4c99f0

sayakpaul mentioned this pull request Apr 7, 2023

Add Karras Sigmas to all schedulers #2905

Closed

sayakpaul reviewed Apr 7, 2023

View reviewed changes

Add test

31da63b

Merge branch 'main' into njindal/2064_1

faf9d8d

patrickvonplaten reviewed Apr 11, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_dpmsolver_multistep.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Apr 11, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_dpmsolver_multistep.py Outdated Show resolved Hide resolved

patrickvonplaten approved these changes Apr 11, 2023

View reviewed changes

sayakpaul and others added 3 commits April 12, 2023 14:18

Apply suggestions from code review

2b8ad6e

Co-authored-by: Patrick von Platen <[email protected]>

fix: repo consistency.

2d9a326

remove Copied from statement from the set_timestep method.

56cc45a

sayakpaul reviewed Apr 12, 2023

View reviewed changes

patrickvonplaten approved these changes Apr 12, 2023

View reviewed changes

sayakpaul and others added 2 commits April 12, 2023 15:58

fix: test

3995af6

Empty commit.

c854dbf

Co-authored-by: njindal <[email protected]>

sayakpaul merged commit 524535b into huggingface:main Apr 12, 2023

Isotr0py mentioned this pull request May 6, 2023

Add use_Karras_sigmas to LMSDiscreteScheduler #3351

Merged

takuma104 mentioned this pull request May 29, 2023

Add support Karras sigmas for StableDiffusionKDiffusionPipeline #2874

Merged

[2064]: Add Karras to DPMSolverMultistepScheduler #3001

[2064]: Add Karras to DPMSolverMultistepScheduler #3001

Uh oh!

Conversation

nipunjindal commented Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nipunjindal commented Apr 6, 2023

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Apr 7, 2023

Uh oh!

nipunjindal commented Apr 7, 2023

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

nipunjindal commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Apr 12, 2023

Uh oh!

sayakpaul commented Apr 12, 2023

Uh oh!

nipunjindal commented Apr 12, 2023

Uh oh!

nipunjindal commented Apr 12, 2023

Uh oh!

sayakpaul commented Apr 12, 2023

Uh oh!

nipunjindal commented Apr 12, 2023

Uh oh!

sayakpaul commented Apr 12, 2023

Uh oh!

Uh oh!

nipunjindal commented Apr 6, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Apr 6, 2023 •

edited

Loading

nipunjindal commented Apr 12, 2023 •

edited

Loading