Feature Request: add SDXL support to animatediff pipeline

[madriss]

Is your feature request related to a problem? Please describe.
Currently the AnimateDiff pipeline only supports SD 1.4 and SD 1.5. AnimateDiff with SDXL seems like a great addition as the quality is greater.

Describe the solution you'd like.
Currently the MotionAdapter cannot load the SDXL motion module :

The following results in an error :

adapter = MotionAdapter.from_pretrained("vladmandic/animatediff-sdxl", torch_dtype=torch.float16)

ValueError: Cannot load <class 'diffusers.models.unet_motion_model.MotionAdapter'> from vladmandic/animatediff-sdxl because the following keys are missing: 
 up_blocks.1.motion_modules.0.norm.bias, down_blocks.2.motion_modules.0.proj_out.bias, up_blocks.1.motion_modules.1.transformer_blocks.0.ff.net.2.bias, down_blocks.3.motion_modules.1.transformer_blocks.0.ff.net.0.proj.weight, 
...

Describe alternatives you've considered.
An alternative solution would be to take inspirtaion from official implementation at https://github.com/guoyww/AnimateDiff
or from https://github.com/s9roll7/animatediff-cli-prompt-travel/tree/main/src/animatediff

Additional context.
Official SDXL motion module : https://huggingface.co/guoyww/animatediff/blob/main/mm_sdxl_v10_beta.ckpt

[DN6]

HI @madriss I think we can look into adding this. @a-r-r-o-w Would you like to give this a try?

[a-r-r-o-w]

HI @madriss I think we can look into adding this. @a-r-r-o-w Would you like to give this a try?

Would love to! I'm already working on it ;)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.