DPM/UniPC schedulers (possibly more) seem to have massive stability issues on model KohakuXL vs ComfyUI

[Xynonners]

Describe the bug

The model KohakuXL in diffusers, seems to run into a massive noise/stability issue that doesn't seem to be present within ComfyUI/AUTO1111, on UniPC/DPM schedulers (possibly more) that doesn't go away with high step counts.
At 15S:

At 30S:

Euler A 30S:

Reproduction

import torch
from accelerate.utils import ProjectConfiguration
from accelerate import Accelerator
from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler
accelerator_config = ProjectConfiguration(
    project_dir="test",
    automatic_checkpoint_naming=True,
    total_limit=10,
)
accelerator = Accelerator(
    log_with="aim",
    mixed_precision="bf16",
    project_config=accelerator_config,
    gradient_accumulation_steps=16,
)
pipeline = StableDiffusionXLPipeline.from_single_file("models/kohakuXLBeta_beta71.safetensors").to(accelerator.device)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
generator = torch.Generator(device=accelerator.device).manual_seed(42)
ims = pipeline(width=1024, height=1024, guidance_scale=14.0, prompt="realistic car 3 d render sci - fi car and sci - fi robotic factory structure in the coronation of napoleon painting and digital billboard with point cloud in the middle, unreal engine 5, keyshot, octane, artstation trending, ultra high detail, ultra realistic, cinematic, 8 k, 1 6 k, in style of zaha hadid, in style of nanospace,", num_inference_steps=15, generator=generator)
for img in ims.images:
    img.save("test.png")

Logs

Not Applicable

System Info

• diffusers version: 0.22.0.dev0
• Platform: Linux-6.5.6-273-tkg-tt-x86_64-with-glibc2.38
• Python version: 3.11.5
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.17.3
• Transformers version: 4.34.1
• Accelerate version: 0.24.0
• xFormers version: not installed
• Using GPU in script?: RTX6000
• Using distributed or parallel set-up in script?: yes in accelerate config, no in this script

Who can help?

@yiyixuxu @patrickvonplaten

[patrickvonplaten]

Can you try whether the fix to DPM provided here: #5541 could help?

[nhnt11]

In my experience, #5541 helps but not completely. I just filed #5689 to document another issue we found.

[Xynonners]

@patrickvonplaten using euler_at_final with either default, use_karras_sigmas, or use_lu_lambdas does help the issue slightly, but it is still very apparent.

use_karras_sigmas tends to be less noisy than default, while use_lu_lambdas tends to have larger but less splotches.

[yiyixuxu]

hi @Xynonners

sorry I can't reproduce this. I tried this script - think euler indeed looks better but I would say dpm output is more or less within the expected range, unlike the output you provided. Did I miss anything here?

import torch
from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler

pipeline = StableDiffusionXLPipeline.from_pretrained("KBlueLeaf/kohaku-xl-beta7.1", torch_dtype=torch.float16)
pipeline.enable_model_cpu_offload()
generator = torch.Generator(device="cuda").manual_seed(42)
ims = pipeline(prompt="realistic car 3 d render sci - fi car and sci - fi robotic factory structure in the coronation of napoleon painting and digital billboard with point cloud in the middle, unreal engine 5, keyshot, octane, artstation trending, ultra high detail, ultra realistic, cinematic, 8 k, 1 6 k, in style of zaha hadid, in style of nanospace,", num_inference_steps=30, generator=generator)
ims.images[0].save("euler.png")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
generator = torch.Generator(device="cuda").manual_seed(42)
ims = pipeline(prompt="realistic car 3 d render sci - fi car and sci - fi robotic factory structure in the coronation of napoleon painting and digital billboard with point cloud in the middle, unreal engine 5, keyshot, octane, artstation trending, ultra high detail, ultra realistic, cinematic, 8 k, 1 6 k, in style of zaha hadid, in style of nanospace,", num_inference_steps=30, use_lu_lambdas =True, euler_at_final = True, generator=generator)

ims.images[0].save("dpm.png")

euler

dpm

[Xynonners]

@yiyixuxu I narrowed it down to the guidance_scale parameter, it seems like DPM is unstable at high guidance scale.

import torch
from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler

pipeline = StableDiffusionXLPipeline.from_single_file("UpDraft/models/kohakuXLBeta_beta71.safetensors", torch_dtype=torch.float16)
pipeline.enable_model_cpu_offload()
generator = torch.Generator(device="cuda").manual_seed(42)
ims = pipeline(guidance_scale=14.0, prompt="realistic car 3 d render sci - fi car and sci - fi robotic factory structure in the coronation of napoleon painting and digital billboard with point cloud in the middle, unreal engine 5, keyshot, octane, artstation trending, ultra high detail, ultra realistic, cinematic, 8 k, 1 6 k, in style of zaha hadid, in style of nanospace,", num_inference_steps=30, generator=generator)
ims.images[0].save("euler.png")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
generator = torch.Generator(device="cuda").manual_seed(42)
ims = pipeline(guidance_scale=14.0, prompt="realistic car 3 d render sci - fi car and sci - fi robotic factory structure in the coronation of napoleon painting and digital billboard with point cloud in the middle, unreal engine 5, keyshot, octane, artstation trending, ultra high detail, ultra realistic, cinematic, 8 k, 1 6 k, in style of zaha hadid, in style of nanospace,", num_inference_steps=30, use_lu_lambdas =True, euler_at_final = True, generator=generator)
ims.images[0].save("dpm.png")

Also, the introduction of accelerate also, while making the model much slower, also seems to serve to destabilize it a larger amount.
dpm:

dpm + accelerate:

[Xynonners]

lowered to 15, seems like euler even has noise issues at this point (cfg 14)
dpm:

euler:

[patrickvonplaten]

@yiyixuxu can you look into this to see if it has been fixed?