IF M breaks

[patrickvonplaten]

Tried DeepFloyd/IF-I-M-v1.0 on nVidia A6000 with 48GB VRAM, it loads ok, but fails on inference:

Exception: CUDA out of memory. Tried to allocate 1536.45 GiB (GPU 0; 47.54 GiB total capacity; 16.54 GiB already allocated; 30.45 GiB free; 16.76 GiB reserved in total by PyTorch)

site-packages/diffusers/models/attention_processor.py:756
756 │ │ attention_probs = attn.get_attention_scores(query, key attention_mask)
site-packages/diffusers/models/attention_processor.py:354 in get_attention_scores
354 │ │ │ baddbmm_input = torch.empty(
355 │ │ │ │ query.shape[0], query.shape[1], key.shape[1], dtype=query.dtype, device=

Then tried to switch from SDP to xFormers and it works, although saying its slow would be an understatement - ~27sec/it on A6000 at 1024x1024.

This looks like something is really wrong with attention layers with DeepFloyd?

Also, there should be a note that DeepFloyd does not support any schedulers except built-in one?

AttributeError: 'FrozenDict' object has no attribute 'variance_type'

[AmericanPresidentJimmyCarter]

Also, there should be a note that DeepFloyd does not support any schedulers except built-in one?

Not true. #3314

Also, there should be a note that DeepFloyd does not support any schedulers except built-in one?

There is an issue. #3280

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[vladmandic]

Not should not mark issue as stale if action is on HF

[patrickvonplaten]

Hmmm I cannot really reproduce the error sadly. The following works just fine for me:

from diffusers import DiffusionPipeline
from diffusers.utils import pt_to_pil
import torch

# stage 1
stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant="fp16", torch_dtype=torch.float16)
stage_1.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
stage_1.enable_model_cpu_offload()

# stage 2
stage_2 = DiffusionPipeline.from_pretrained(
    "DeepFloyd/IF-II-M-v1.0", text_encoder=None, variant="fp16", torch_dtype=torch.float16
)
stage_2.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
stage_2.enable_model_cpu_offload()

prompt = 'a photo of a kangaroo wearing an orange hoodie and blue sunglasses standing in front of the eiffel tower holding a sign that says "very deep learning"'

# text embeds
prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt)

generator = torch.manual_seed(0)

image = stage_1(prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt").images
pt_to_pil(image)[0].save("./if_stage_I.png")

image = stage_2(
    image=image, prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt"
).images
pt_to_pil(image)[0].save("./if_stage_II.png")

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.