Suggestion to add TEs

[janekm]

Did you like the remote VAE solution?

I do... it's helpful when training/finetuning models, makes the infrastructure easier to set up.

What can be improved about the current solution?

Adding TEs would be very helpful for a complete solution for auxiliary model off-loading during training.
Consider caching, then it could further simplify model fine-tuning setups.

What other VAEs you would like to see if the pilot goes well?

StepVideo VAE: https://huggingface.co/stepfun-ai/stepvideo-t2v/tree/main/vae
HunYuan CLIP (for Stepvideo): https://huggingface.co/stepfun-ai/stepvideo-t2v/tree/main/hunyuan_clip
TE for LuminaVision2: Gemma-2-2B

Notify the members of the team

@hlky @sayakpaul

[hlky]

Hi @janekm. Thanks for sharing your use case and feedback.

We are exploring the possibility of adding text encoders, also VAE encode which also has application in training/fine-tuning, and image encoders for IPAdapter as an example.

Currently no data is stored or tracked, there would be privacy concerns for caching so we will consider the options.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.