Skip to content

Add UFOGenScheduler to Community Examples #6650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions examples/community/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ If a community doesn't work as expected, please open an issue and ping the autho
| AnimateDiff Image-To-Video Pipeline | Experimental Image-To-Video support for AnimateDiff (open to improvements) | [AnimateDiff Image To Video Pipeline](#animatediff-image-to-video-pipeline) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://drive.google.com/file/d/1TvzCDPHhfFtdcJZe4RLloAwyoLKuttWK/view?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |
| IP Adapter FaceID Stable Diffusion | Stable Diffusion Pipeline that supports IP Adapter Face ID | [IP Adapter Face ID](#ip-adapter-face-id) | - | [Fabio Rigano](https://github.com/fabiorigano) |
| InstantID Pipeline | Stable Diffusion XL Pipeline that supports InstantID | [InstantID Pipeline](#instantid-pipeline) | [![Hugging Face Space](https://img.shields.io/badge/🤗%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/InstantX/InstantID) | [Haofan Wang](https://github.com/haofanwang) |
| UFOGen Scheduler | Scheduler for UFOGen Model (compatible with Stable Diffusion pipelines) | [UFOGen Scheduler](#ufogen-scheduler) | - | [dg845](https://github.com/dg845) |

To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.

Expand Down Expand Up @@ -3605,3 +3606,32 @@ image = pipe(
controlnet_conditioning_scale=0.8,
).images[0]
```

### UFOGen Scheduler

[UFOGen](https://arxiv.org/abs/2311.09257) is a generative model designed for fast one-step text-to-image generation, trained via adversarial training starting from an initial pretrained diffusion model such as Stable Diffusion. `scheduling_ufogen.py` implements a onestep and multistep sampling algorithm for UFOGen models compatible with pipelines like `StableDiffusionPipeline`. A usage example is as follows:

```py
import torch
from diffusers import StableDiffusionPipeline

from scheduling_ufogen import UFOGenScheduler

# NOTE: currently, I am not aware of any publicly available UFOGen model checkpoints trained from SD v1.5.
ufogen_model_id_or_path = "/path/to/ufogen/model"
pipe = StableDiffusionPipeline(
ufogen_model_id_or_path,
torch_dtype=torch.float16,
)

# You can initialize a UFOGenScheduler as follows:
pipe.scheduler = UFOGenScheduler.from_config(pipe.scheduler.config)

prompt = "Three cats having dinner at a table at new years eve, cinematic shot, 8k."

# Onestep sampling
onestep_image = pipe(prompt, num_inference_steps=1).images[0]

# Multistep sampling
multistep_image = pipe(prompt, num_inference_steps=4).images[0]
```
Loading