Skip to content

Please create simple way to load new embeddings on Stable Diffusion Pipeline #1985

Closed
@piEsposito

Description

@piEsposito

Is your feature request related to a problem? Please describe.
It is frustrating to load Textual Inversion embeddings on SD pipelines, adds bloat to the code. It would be good if we could just pipeline.load_embeddings and add new embeddings ad-hoc to the pipeline.

Describe the solution you'd like
I would like to implement a method on Stable Diffusion pipelines to let people load_embeddings and append them to ones from the text encoder and tokenizer, something like:

  • pipeline.load_embeddings({"emb1": "emb1.ckpt"}), then
  • Embedding is loaded and appended to the embedding matrix of text encoder. Token is added to tokenizer.
  • Predictions work with the new token smoothly.

We could even hack a way to load tokens with multiple embedding columns, alike to what we can do on Auto1111.

Describe alternatives you've considered

  • Do that by hand every single time, but then if would bloat my code and wouldn't benefit the Diffusers community
  • Don't use textual inversion at all, but this would be no-fun.

Additional context
I can implement it and open a PR if you approve the idea.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions