Skip to content

Commit 978dec9

Browse files
a-r-r-o-wEdoardoBottapatrickvonplaten
authored
[Community] AnimateDiff + Controlnet Pipeline (#5928)
* begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <[email protected]> * update Co-Authored-By: EdoardoBotta <[email protected]> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <[email protected]> Co-authored-by: Patrick von Platen <[email protected]>
1 parent 79a7ab9 commit 978dec9

File tree

2 files changed

+1202
-0
lines changed

2 files changed

+1202
-0
lines changed

‎examples/community/README.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ prompt-to-prompt | change parts of a prompt and retain image structure (see [pap
5050
| Latent Consistency Interpolation Pipeline | Interpolate the latent space of Latent Consistency Models with multiple prompts | [Latent Consistency Interpolation Pipeline](#latent-consistency-interpolation-pipeline) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1pK3NrLWJSiJsBynLns1K1-IDTW9zbPvl?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |
5151
| Regional Prompting Pipeline | Assign multiple prompts for different regions | [Regional Prompting Pipeline](#regional-prompting-pipeline) | - | [hako-mikan](https://github.com/hako-mikan) |
5252
| LDM3D-sr (LDM3D upscaler) | Upscale low resolution RGB and depth inputs to high resolution | [StableDiffusionUpscaleLDM3D Pipeline](https://github.com/estelleafl/diffusers/tree/ldm3d_upscaler_community/examples/community#stablediffusionupscaleldm3d-pipeline) | - | [Estelle Aflalo](https://github.com/estelleafl) |
53+
| AnimateDiff ControlNet Pipeline | Combines AnimateDiff with precise motion control using ControlNets | [AnimateDiff ControlNet Pipeline](#animatediff-controlnet-pipeline) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1SKboYeGjEQmQPWoFC0aLYpBlYdHXkvAu?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) and [Edoardo Botta](https://github.com/EdoardoBotta) |
5354
| DemoFusion Pipeline | Implementation of [DemoFusion: Democratising High-Resolution Image Generation With No $$$](https://arxiv.org/abs/2311.16973) | [DemoFusion Pipeline](#DemoFusion) | - | [Ruoyi Du](https://github.com/RuoyiDu) |
5455

5556
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
@@ -2839,6 +2840,70 @@ The Pipeline supports `compel` syntax. Input prompts using the `compel` structur
28392840
* Reconstructed image:
28402841
* ![dps_generated_image](https://github.com/tongdaxu/Images/assets/22267548/b74f084d-93f4-4845-83d8-44c0fa758a5f)
28412842
2843+
### AnimateDiff ControlNet Pipeline
2844+
2845+
This pipeline combines AnimateDiff and ControlNet. Enjoy precise motion control for your videos! Refer to [this](https://github.com/huggingface/diffusers/issues/5866) issue for more details.
2846+
2847+
```py
2848+
import torch
2849+
from diffusers import AutoencoderKL, ControlNetModel, MotionAdapter
2850+
from diffusers.pipelines import DiffusionPipeline
2851+
from diffusers.schedulers import DPMSolverMultistepScheduler
2852+
from PIL import Image
2853+
2854+
motion_id = "guoyww/animatediff-motion-adapter-v1-5-2"
2855+
adapter = MotionAdapter.from_pretrained(motion_id)
2856+
controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=torch.float16)
2857+
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", torch_dtype=torch.float16)
2858+
2859+
model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
2860+
pipe = DiffusionPipeline.from_pretrained(
2861+
model_id,
2862+
motion_adapter=adapter,
2863+
controlnet=controlnet,
2864+
vae=vae,
2865+
custom_pipeline="pipeline_animatediff_controlnet",
2866+
).to(device="cuda", dtype=torch.float16)
2867+
pipe.scheduler = DPMSolverMultistepScheduler.from_pretrained(
2868+
model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1
2869+
)
2870+
pipe.enable_vae_slicing()
2871+
2872+
conditioning_frames = []
2873+
for i in range(1, 16 + 1):
2874+
conditioning_frames.append(Image.open(f"frame_{i}.png"))
2875+
2876+
prompt = "astronaut in space, dancing"
2877+
negative_prompt = "bad quality, worst quality, jpeg artifacts, ugly"
2878+
result = pipe(
2879+
prompt=prompt,
2880+
negative_prompt=negative_prompt,
2881+
width=512,
2882+
height=768,
2883+
conditioning_frames=conditioning_frames,
2884+
num_inference_steps=12,
2885+
).frames[0]
2886+
2887+
from diffusers.utils import export_to_gif
2888+
export_to_gif(result.frames[0], "result.gif")
2889+
```
2890+
2891+
<table>
2892+
<tr><td colspan="2" align=center><b>Conditioning Frames</b></td></tr>
2893+
<tr align=center>
2894+
<td align=center><img src="https://user-images.githubusercontent.com/7365912/265043418-23291941-864d-495a-8ba8-d02e05756396.gif" alt="input-frames"></td>
2895+
</tr>
2896+
<tr><td colspan="2" align=center><b>AnimateDiff model: SG161222/Realistic_Vision_V5.1_noVAE</b></td></tr>
2897+
<tr>
2898+
<td align=center><img src="https://github.com/huggingface/diffusers/assets/72266394/baf301e2-d03c-4129-bd84-203a1de2b2be" alt="gif-1"></td>
2899+
<td align=center><img src="https://github.com/huggingface/diffusers/assets/72266394/9f923475-ecaf-452b-92c8-4e42171182d8" alt="gif-2"></td>
2900+
</tr>
2901+
<tr><td colspan="2" align=center><b>AnimateDiff model: CardosAnime</b></td></tr>
2902+
<tr>
2903+
<td align=center><img src="https://github.com/huggingface/diffusers/assets/72266394/b2c41028-38a0-45d6-86ed-fec7446b87f7" alt="gif-1"></td>
2904+
<td align=center><img src="https://github.com/huggingface/diffusers/assets/72266394/eb7d2952-72e4-44fa-b664-077c79b4fc70" alt="gif-2"></td>
2905+
</tr>
2906+
</table>
28422907
### DemoFusion
28432908
This pipeline is the official implementation of [DemoFusion: Democratising High-Resolution Image Generation With No $$$](https://arxiv.org/abs/2311.16973).
28442909
The original repo can be found at [repo](https://github.com/PRIS-CV/DemoFusion).

0 commit comments

Comments
 (0)