Skip to content

Artifacts with DPM++ 2M SDE Karras, even when using use_lu_lambdas #6295

Closed
@ivanprado

Description

@ivanprado

Describe the bug

The results obtained using DPM++ 2M SDE Karras contain artifacts that suggest there is some bug. This happens across different XL models, although is not so visible in the base SDXL. Using Automatic1111 with the same models and the same kind of scheduler results in high-quality results, and it seems impossible to achieve the same quality using the diffusers equivalent.

An image generated using Automatic1111 DPM++ 2M SDE Karras with the Juggernaut XL v6 model
adorable_infant_dpm++_2M_SDE_Karras_jugger6xl

An image generated using the equivalent scheduler from diffusers:
adorable_infant_dpm++_2M_SDE_Karras_jugger6xl_diffusers

The artifacts are very visible in the last image.

The following didn't help:

  • Loading the model in fp32
  • use_karras_sigmas=False
  • use_karras_sigmas=False and dpmsolver++
  • euler_at_final=True and use_lu_lambdas=True

I see this happening clearly at least with the following models:

  • frankjoshua/juggernautXL_version6Rundiffusion
  • Lykon/dreamshaper-xl-1-0

Reproduction

I've created a github repo to help reproducing the problem. Also, a simpler code snippet is provided below:

from diffusers import AutoPipelineForText2Image
from diffusers.schedulers import DPMSolverMultistepScheduler
import torch

model_id = "frankjoshua/juggernautXL_version6Rundiffusion"
pipeline = AutoPipelineForText2Image.from_pretrained(
    model_id, 
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
).to("cuda")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
        pipeline.scheduler.config, 
        use_karras_sigmas=True,
        sde_type="sde-dpmsolver++",
        euler_at_final=True,
        use_lu_lambdas=True
        )

prompt = "Adorable infant playing with a variety of colorful rattle toys."

results = pipeline(
    prompt=prompt, 
    guidance_scale=3,
    generator=torch.Generator(device="cuda").manual_seed(42),  
    num_inference_steps=25, 
    height=768, 
    width=1344)
display(results.images[0])

Logs

No response

System Info

  • diffusers version: 0.24.0
  • Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.31
  • Python version: 3.10.13
  • PyTorch version (GPU?): 2.1.1 (True)
  • Huggingface_hub version: 0.20.1
  • Transformers version: 4.36.2
  • Accelerate version: 0.25.0
  • xFormers version: not installed
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: no

Who can help?

@yiyixuxu @patrickvonplaten

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions