Why do Diffusers schedulers produce lower quality outputs compared to ComfyUI?

[nducthang]

Discussed in #8682

^{Originally posted by nducthang June 24, 2024}
Hi,

I'm encountering an issue when comparing the quality of ComfyUI and Diffusers. I've noticed that the output of Diffusers is consistently lower than ComfyUI in many cases, despite using the same settings and seed. For the base Diffusers, I've utilized: https://github.com/huggingface/diffusers/blob/main/examples/community/lpw_stable_diffusion_xl.py.

Upon closer inspection, I've identified differences in the scheduler/ksampler between the two base codes. I've also observed variations in CLIP Embedding between the two base codes, but in my experiments, this hasn't significantly impacted the output. The main issue seems to lie with the KSampler.

Has anyone else encountered this issue or have any ideas on improving the Scheduler algorithm of Diffusers?

Here are some prompts I've experimented:
Model: RVXL - Size: (896, 1152)
Positive prompt:

female, attractive woman, pretty middle-aged woman, thick hair, (((Caucasian, European, Scandinavian female))), ((hazel eyes, HazelEyed)). (Brunette (Light-Brown-Hair)), ((((long rectangular face, elongated face, oblong face shape, angular chiseled face)), ((wide jaw, big strong chin)))). (((1980s magazine advertisement. Living room. CRT Televesion. 1980s aesthetic. 1980s interior design.))) [object Object] . high quality, dim lighting, soft lighting, sharp focus, f5.6, dslr, High Detail, detailed, ((wide shot))

Negative prompt:

(((male))), (small chin, receding-chin, puffy face), (((Asian, Chinese, Korean, Japanese, Indian, Pakistani, Black, African, Persian, Arab, Middle Eastern, Hispanic, Latino))), (small chin, receding-chin, puffy face), (blurry), (BadDream:1.2), (UnrealisticDream:1.2), ((bad-hands-5)), (strabismus, cross-eyed:1.2), (signature, watermark, name), (worst quality, poor quality, low quality), ((deformed)), (extra limbs), (extra arms), (extra legs), disfigured, malformed, (nude:1.4), (naked:1.4), (nsfw:1.4), (bikini:1.4), (lingerie:1.4), (underwear:1.4), (teen:1.4), (tween:1.4), (teenage:1.4), (kid:1.6), (child:1.6), (topless, shirtless:1.4), (((greyscale))), (cleavage:1.2), (nipples:1.4)

[sayakpaul]

Please provide the diffusers code you're using so that we can guide you accordingly. Cc: @asomoza

[asomoza]

Also some example images with what you consider a good result (automatic1111) and a bad result (diffusers) but please can you change that prompt to not being human beings.

[nducthang]

@sayakpaul @asomoza Thank you for responses.
This is code that I used:

from lpw_stable_diffusion_xl import SDXLLongPromptWeightingPipeline
import torch
from diffusers import DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler


SD_XL_REALVISXL_V4_0_MODEL_ID = "SG161222/RealVisXL_V4.0"
prompt = "female, attractive woman, pretty middle-aged woman, thick hair, (((Caucasian, European, Scandinavian female))), ((hazel eyes, HazelEyed)). (Brunette (Light-Brown-Hair)), ((((long rectangular face, elongated face, oblong face shape, angular chiseled face)), ((wide jaw, big strong chin)))). (1980s magazine advertisement. Living room. CRT Televesion. 1980s aesthetic. 1980s interior design.) [object Object] . high quality, dim lighting, soft lighting, sharp focus, f5.6, dslr, High Detail, detailed, ((wide shot))"
negative_prompt = "(((male))), (small chin, receding-chin, puffy face), (((Asian, Chinese, Korean, Japanese, Indian, Pakistani, Black, African, Persian, Arab, Middle Eastern, Hispanic, Latino))), (small chin, receding-chin, puffy face), (blurry), (BadDream:1.2), (UnrealisticDream:1.2), ((bad-hands-5)), (strabismus, cross-eyed:1.2), (signature, watermark, name), (worst quality, poor quality, low quality), ((deformed)), (extra limbs), (extra arms), (extra legs), disfigured, malformed, (nude:1.4), (naked:1.4), (nsfw:1.4), (bikini:1.4), (lingerie:1.4), (underwear:1.4), (teen:1.4), (tween:1.4), (teenage:1.4), (kid:1.6), (child:1.6), (topless, shirtless:1.4), (((greyscale))), (cleavage:1.2), (nipples:1.4)"

pipe = SDXLLongPromptWeightingPipeline.from_pretrained(
    SD_XL_REALVISXL_V4_0_MODEL_ID,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
).to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
# pipe.schduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
# for seed in [0, 5, 100, 1024]:
for seed in [24483]:
    torch.manual_seed(seed)
    generator = [torch.Generator(device="cuda").manual_seed(seed)]
    image = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        generator=generator,
        num_inference_steps=30,
        denoising_end=1.0,
        guidance_scale=3.5,
        width=832,
        height=1216,
        output_type="pil",
    ).images[0]

    image.save(f"RVXL-{seed}.png")

Results of Diffusers and ComfyUI:


Somtime, if change seeds, Diffusers base can be generate a TV:

With ComfyUI I see stable results and this phenomenon does not occur.

[nducthang]

Sometimes images get corrupted (random seed) in diffusers base. I tried change scheduler but the result are not improved

[asomoza]

is there a reason why you're using such a low cfg?

[nducthang]

is there a reason why you're using such a low cfg?

When I tested before and decided to use DPMSolverMultistepScheduler Karras I tested a cfg of 3.5 and 7. The 3.5 seemed to give better results than 7. 7 started causing there to be artifacts even at 40 steps. 30 steps with a cfg of 3.5 seemed to be the sweet spot.

[asomoza]

ok, I see that you're getting some weird results also, I'll do some tests but this doesn't seem to be an issue with the schedulers, I can generate just fine with your prompt (truncated) and without artifacts. I'll test with the long weighting prompts.

With DPMSolverMultistepScheduler I always use a cfg of 6.5 and 25 steps and works fine, so its weird you're getting artifacts with more steps.

This reminds me an issue we had with the schedulers way back, are you using the latest release?

[nducthang]

ok, I see that you're getting some weird results also, I'll do some tests but this doesn't seem to be an issue with the schedulers, I can generate just fine with your prompt (truncated) and without artifacts. I'll test with the long weighting prompts.

With DPMSolverMultistepScheduler I always use a cfg of 6.5 and 25 steps and works fine, so its weird you're getting artifacts with more steps.

This reminds me an issue we had with the schedulers way back, are you using the latest release?

Yes, I also find that the diffusers can be stable with shorter prompt and without weighting. However, If I evaluate emotionally, I think the quality doesn't seem to be as good as ComfyUI (Same all settings). So, I think schedulers might be the cause.

In my experiment, I am using lastest release.

[asomoza]

I'm definitely not getting your bad results, this is the kind I get:

I did lots and never got the artifacts or the weird television thing.

CFG: 6.5
steps: 25

[asomoza]

Also I used the new library by the same author @xhinker as the pipeline:

https://github.com/xhinker/sd_embed

But the results should be the same I think, if not, maybe you can try using that.

[asomoza]

Also do I need to use something in comfyui to be able to use those prompts? If I just use the vanilla nodes I get worse results than diffusers in my opinion, they look kinda fake and plastic with the exact same params:

comfyui	diffusers

[nducthang]

@asomoza Do you used SDXLLongPromptWeightingPipeline from https://github.com/huggingface/diffusers/blob/main/examples/community/lpw_stable_diffusion_xl.py ?

Seed 0, 100 and 1024 (CFG: 3.5) (Same results with CFG 6.5):

If use choose width=1024 and height=1536. The results will be worse.

[asomoza]

Do you used SDXLLongPromptWeightingPipeline

No, I used the library I posted which is from the same author which enables the use of long weighted prompts without the need to load any specific pipeline.

Also take into account that you can't just use any resolution, to get the best results you'll need to use the resolution the model was trained before, and mostly they recommend that they match the number of pixels of 1024x1024, so if you go high with one it's better to lower the other one.

1024x1536 goes way higher than that, I can get good results with it though but I also start getting bad ones, but the same happens with any UI, is something related to the model.

As I answered in the other thread, this is mostly something to do with your environment or something you're doing, the bad quality is not something related to the schedulers or diffusers. So we just need to find what is making you get those bad results.

Maybe you can post your environment, you can get it with diffusers-cli env

[nducthang]

I attached my workflow in ComfyUI. If look kinda fake and plastic, you can use CFG 3.5.
workflow (2).json

The strange thing is that I find the pipeline diffusers I use give unstable results. I will try the library https://github.com/xhinker/sd_embed that you suggested to see if the quality improves, thank you!

[xhinker]

I tried your prompt.
Using

import gc
import torch
from diffusers import StableDiffusionXLPipeline
from sd_embed.embedding_funcs import get_weighted_text_embeddings_sdxl
from diffusers.utils import make_image_grid

model_path = "SG161222/RealVisXL_V4.0"
pipe = StableDiffusionXLPipeline.from_pretrained(
    model_path,
    torch_dtype=torch.float16
)

#%%
pipe.to('cuda')
prompt = """\
female, attractive woman, pretty middle-aged woman, thick hair, (((Caucasian, European, Scandinavian female))), ((hazel eyes, HazelEyed)). (Brunette (Light-Brown-Hair)), ((((long rectangular face, elongated face, oblong face shape, angular chiseled face)), ((wide jaw, big strong chin)))). (((1980s magazine advertisement. Living room. CRT Televesion. 1980s aesthetic. 1980s interior design.))) [object Object] . high quality, dim lighting, soft lighting, sharp focus, f5.6, dslr, High Detail, detailed, ((wide shot))
"""

neg_prompt = """\
(((male))), (small chin, receding-chin, puffy face), (((Asian, Chinese, Korean, Japanese, Indian, Pakistani, Black, African, Persian, Arab, Middle Eastern, Hispanic, Latino))), (small chin, receding-chin, puffy face), (blurry), (BadDream:1.2), (UnrealisticDream:1.2), ((bad-hands-5)), (strabismus, cross-eyed:1.2), (signature, watermark, name), (worst quality, poor quality, low quality), ((deformed)), (extra limbs), (extra arms), (extra legs), disfigured, malformed, (nude:1.4), (naked:1.4), (nsfw:1.4), (bikini:1.4), (lingerie:1.4), (underwear:1.4), (teen:1.4), (tween:1.4), (teenage:1.4), (kid:1.6), (child:1.6), (topless, shirtless:1.4), (((greyscale))), (cleavage:1.2), (nipples:1.4)
"""

(
    prompt_embeds
    , prompt_neg_embeds
    , pooled_prompt_embeds
    , negative_pooled_prompt_embeds
) = get_weighted_text_embeddings_sdxl(
    pipe
    , prompt = prompt
    , neg_prompt = neg_prompt
)

image = pipe(
    prompt_embeds                   = prompt_embeds
    , negative_prompt_embeds        = prompt_neg_embeds
    , pooled_prompt_embeds          = pooled_prompt_embeds
    , negative_pooled_prompt_embeds = negative_pooled_prompt_embeds
    , num_inference_steps           = 30
    , height                        = 1024 
    , width                         = 1024
    , guidance_scale                = 7.0
    , generator                     = torch.Generator("cuda").manual_seed(2)
    , num_images_per_prompt         = 4
).images
# display(image)

del prompt_embeds, prompt_neg_embeds,pooled_prompt_embeds, negative_pooled_prompt_embeds
pipe.to('cpu')
gc.collect()
torch.cuda.empty_cache()

make_image_grid(image,rows=2, cols=2)

I got: