Skip to content

Remote VAE encode #11069

Closed
Closed
@vladmandic

Description

@vladmandic

Describe the bug

Remote VAE encode was added via PR ##11017
I've added remote-vae feature for img2img and inpaint workflows for sd15, sdxl and flux.1

  • sdxl: works for both img2img and inpaint
  • sd15: works for img2img, but fails for inpaint
    • binary mask (typically a pil image type l) cannot be combined with image param if its a tensor
    • using remote vae to also encode mask image does work, but results in nonsense output as its not designed to encode binary masks
  • flux: fails for img2img and inpaint
    • flux pipeline doesn't care if input image param is already encoded, it tries to encode it again
/home/vlado/dev/sdnext/venv/lib/python3.12/site-packages/diffusers/pipelines/flux/pipeline_flux_img2img.py:637 in prepare_latents636 │   │   image = image.to(device=device, dtype=dtype)
│ ❱  637 │   │   image_latents = self._encode_vae_image(image=image, generator=generator)

Reproduction

(see above)

Logs

System Info

diffusers main branch commit 82188c

Who can help?

@hlky

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleIssues that haven't received updates

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions