Skip to content

fix DPM Scheduler with use_karras_sigmas option #6477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Jan 19, 2024
Merged

fix DPM Scheduler with use_karras_sigmas option #6477

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
9db1b45
fix
Jan 7, 2024
72e58ba
fix copies
Jan 7, 2024
9a49479
refactor
Jan 11, 2024
6fbdd12
fix dpm inverse
Jan 15, 2024
1b2a428
dpm singlestep
Jan 15, 2024
c6ec05f
add valueerror for algorithems that does not support denoise_to_zero
Jan 15, 2024
aa9d57c
Apply suggestions from code review
yiyixuxu Jan 16, 2024
15d4fb3
remove dpmsolver and sde-dpmsolver
Jan 17, 2024
1827a4c
udpate and deprecate inverse
Jan 17, 2024
3afde1f
merge
Jan 18, 2024
4271d2f
reverse changes to inverse scheduler
Jan 18, 2024
8d54850
fix
Jan 18, 2024
b23cade
remove file
Jan 18, 2024
a2e0903
deprecate dpmsolver from dpm singlestep
Jan 18, 2024
c8e9f8b
Update src/diffusers/schedulers/__init__.py
yiyixuxu Jan 18, 2024
d084322
Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
yiyixuxu Jan 18, 2024
49b8e80
Revert "reverse changes to inverse scheduler"
Jan 18, 2024
179acd7
Revert "udpate and deprecate inverse"
Jan 18, 2024
8b16ab6
Revert "remove dpmsolver and sde-dpmsolver"
Jan 18, 2024
41e25e9
fix
Jan 18, 2024
085498d
dpm inverse
Jan 18, 2024
4666722
fix
Jan 18, 2024
b68106d
fix deprecate
Jan 18, 2024
91c02fd
fix tests
Jan 18, 2024
643cdc1
style
Jan 18, 2024
64e35fd
fix format
Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ def set_timesteps(self, num_inference_steps: int = None, device: Union[str, torc
sigmas = np.flip(sigmas).copy()
sigmas = self._convert_to_karras(in_sigmas=sigmas, num_inference_steps=num_inference_steps)
timesteps = np.array([self._sigma_to_t(sigma, log_sigmas) for sigma in sigmas]).round()
sigmas = np.concatenate([sigmas, sigmas[-1:]]).astype(np.float32)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patrickvonplaten

Was there a reason to set the sigmas this way? or is it just a mistake that we didn't catch in the PR?

By repeating the last sigma twice, we make the very last step a dummy step with zero step size

i.e. this code here is x_t= sample because we have sigma_t = sigma_s0 and h=0

x_t = (
                    (sigma_t / sigma_s0) * sample
                    - (alpha_t * (torch.exp(-h) - 1.0)) * D0
                    - 0.5 * (alpha_t * (torch.exp(-h) - 1.0)) * D1
                )

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

Line 630 in 774f5c4

if self.config.solver_type == "midpoint":

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! Was this the culprit?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this was it!

sigmas = np.concatenate([sigmas, np.array([0])]).astype(np.float32)
elif self.config.use_lu_lambdas:
lambdas = np.flip(log_sigmas.copy())
lambdas = self._convert_to_lu(in_lambdas=lambdas, num_inference_steps=num_inference_steps)
Expand Down Expand Up @@ -831,7 +831,9 @@ def step(

# Improve numerical stability for small number of steps
lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
self.config.euler_at_final or (self.config.lower_order_final and len(self.timesteps) < 15)
self.config.euler_at_final
or (self.config.lower_order_final and len(self.timesteps) < 15)
or self.config.use_karras_sigmas
)
lower_order_second = (
(self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15
Expand Down
4 changes: 3 additions & 1 deletion src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -824,7 +824,9 @@ def step(

# Improve numerical stability for small number of steps
lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
self.config.euler_at_final or (self.config.lower_order_final and len(self.timesteps) < 15)
self.config.euler_at_final
or (self.config.lower_order_final and len(self.timesteps) < 15)
or self.config.use_karras_sigmas
)
lower_order_second = (
(self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15
Expand Down