AnimateDiffPipeline: add ImageToVideo and VideoToVideo

[jon-chuang]

Can adapt the work in https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved

[jon-chuang]

Seems pretty simple, just need to add a prepare_latents which initializes the latents to encoded image.

A workaround right now is to encode the images in advance, and pass them in as latents param.

[jon-chuang]

Btw, you should probably also check that latents shape is correct in the original pipeline.

[a-r-r-o-w]

Hey @jon-chuang. ImageToVideo should be an easy addition to the current implementation, I think, and I can take it up. For VideoToVideo, do you mean the controlnet example in the linked repo (#5866 is relevant in that case)?

[jon-chuang]

@a-r-r-o-w I have something barely working right now, I might upstream it at later date. I think the ComfyUI has a lot of hacks to get it to work well, let me try to recreate them.

[a-r-r-o-w]

Hey @jon-chuang. Do you have any updates on the ImageToVideo addition and do you plan on opening a PR for the same? If not, I'll be happy to start looking into it.

[jon-chuang]

Hello Aryan, sure go ahead, I'll post some code here soon in case it will help you.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.