DPMSolverSinglestepScheduler with use_karras_sigmas=True triggers RuntimeError: a Tensor with 2 elements cannot be converted to Scalar

[alexisrolland]

Describe the bug

Hello @patrickvonplaten and @williamberman 🤗

Running StableDiffusionPipeline with DPMSolverSinglestepScheduler and use_karras_sigmas=True triggers the error RuntimeError: a Tensor with 2 elements cannot be converted to Scalar when generating images with a high num_inference_steps (~100). Note this issue does not happen with low num_inference_steps (~20).

Reproduction

import os
import torch
from diffusers import StableDiffusionPipeline, DPMSolverSinglestepScheduler

MODEL_PATH = os.getenv('MODEL_PATH')
pipeline = StableDiffusionPipeline.from_pretrained(MODEL_PATH, torch_dtype=torch.float16, variant='fp16')
pipeline.scheduler = DPMSolverSinglestepScheduler.from_pretrained(MODEL_PATH, subfolder='scheduler', use_karras_sigmas=True)
pipeline.enable_model_cpu_offload()

prompts = ["a cat playing with a ball in the forest"]
negative_prompts =["bad quality"]
seed = 123456
generators = torch.Generator(device='cuda').manual_seed(seed)

image = pipeline(
    prompt=prompts,
    negative_prompt=negative_prompts,
    width=800,
    height=1200,
    num_inference_steps=100,
    guidance_scale=6,
    num_images_per_prompt=1,
    generator=generators
).images[0].save("image.png")

Logs

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 688, in __call__
    latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]
  File "/usr/local/lib/python3.10/site-packages/diffusers/schedulers/scheduling_dpmsolver_singlestep.py", line 670, in step
    step_index = step_index.item()
RuntimeError: a Tensor with 2 elements cannot be converted to Scalar

System Info

• diffusers version: 0.19.3
• Platform: Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.36
• Python version: 3.10.12
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Huggingface_hub version: 0.16.4
• Transformers version: 4.30.2
• Accelerate version: 0.21.0
• xFormers version: 0.0.20
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

No response

[sayakpaul]

Cc: @yiyixuxu too.

[yiyixuxu]

thanks for the issue. It is caused by duplicated timesteps and will be fixed by #4187
(I will start work on this today!)

[CodeCorrupt]

Hey, I wanted to bump this issue since it seems the linked PR is for DPMSolverMultistepScheduler and I'm currently seeing the issue with DPMSolverSinglestepScheduler when use_karras_sigmas=True.
Also of note is that I'm only seeing the error with SD1.5 pipelines and when the step count is > ~10 steps. SDXL pipeline appears to be working as expected.

[patrickvonplaten]

Linked PR: #4986

[yiyixuxu]

fixed with #4986

[Z-YuPeng]

DPMSolverSinglestepScheduler when use_karras_sigmas=True still has this bug in the latest diffuser version 0.21.4 .

[patrickvonplaten]

cc @yiyixuxu could you check here?

[alexisrolland]

I confirm this is fixed for both DPMSolverSinglestepScheduler and DPMSolverMultistepScheduler as I've tested this again on 0.22.2.

[burgalon]

This is still happening with 0.23.1 RuntimeError: a Tensor with 2 elements cannot be converted to Scalar

cc @yiyixuxu

Does not reproduce when pip install git+https://github.com/huggingface/diffusers@ed507680e35b7628bc11235255cfc58ad1101626

Here's some code to reproduce

import torch
from diffusers import StableDiffusionPipeline, StableDiffusionControlNetInpaintPipeline, DPMSolverMultistepScheduler, ControlNetModel
from diffusers.utils import load_image
control = ControlNetModel.from_pretrained(torch_dtype=torch.float16, pretrained_model_name_or_path="lllyasviel/control_v11p_sd15_openpose").to("cuda")
base = StableDiffusionPipeline.from_pretrained("/data/models/690204-sd15", torch_dtype=torch.float16).to("cuda")
pipe = StableDiffusionControlNetInpaintPipeline(
    vae=base.vae,
    text_encoder=base.text_encoder,
    tokenizer=base.tokenizer,
    unet=base.unet,
    scheduler=base.scheduler,
    safety_checker=None,
    feature_extractor=base.feature_extractor,
    controlnet=control,
).to('cuda')
scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, algorithm_type="sde-dpmsolver++", use_karras_sigmas=True)
pipe.scheduler = scheduler
image = load_image('https://sdbooth2-production.s3.amazonaws.com/pph45nwtz5w0afa8nlhbt41djwep')
pipe(prompt="test", image=image, mask_image=image, control_image=image)

exception

File /opt/venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_dpmsolver_multistep.py:888, in DPMSolverMultistepScheduler.add_noise(self, original_samples, noise, timesteps)
    885     schedule_timesteps = self.timesteps.to(original_samples.device)
    886     timesteps = timesteps.to(original_samples.device)
--> 888 step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps]
    890 sigma = sigmas[step_indices].flatten()
    891 while len(sigma.shape) < len(original_samples.shape):

File /opt/venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_dpmsolver_multistep.py:888, in <listcomp>(.0)
    885     schedule_timesteps = self.timesteps.to(original_samples.device)
    886     timesteps = timesteps.to(original_samples.device)
--> 888 step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps]
    890 sigma = sigmas[step_indices].flatten()
    891 while len(sigma.shape) < len(original_samples.shape):

RuntimeError: a Tensor with 2 elements cannot be converted to Scalar```

[patrickvonplaten]

@yiyixuxu can you take a look at: #4631 (comment) ?

@burgalon can you please make sure to only use models & images that can be publicly accessed ?
I can't use "/data/models/690204-sd15" or 'https://sdbooth2-production.s3.amazonaws.com/pph45nwtz5w0afa8nlhbt41djwep'