[WIP] test prepare_latents for ltx0.95

[yiyixuxu]

for #10968

import torch
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXConditionPipeline, LTXVideoCondition
from diffusers.utils import export_to_video, load_video


device = "cuda:2"
dtype = torch.bfloat16

# command to convert the checkpoint
"""
python scripts/convert_ltx_to_diffusers.py --transformer_ckpt_path "./ltx-video-2b-v0.9.5rc1.safetensors" --vae_ckpt_path "./ltx-video-2b-v0.9.5rc1.safetensors" --output_path "/raid/yiyi/LTX-Video-95" --version 0.9.5 --save_pipeline
"""


repo = "/raid/yiyi/LTX-Video-95"

# Initialize the pipeline
pipe = LTXConditionPipeline.from_pretrained(repo, torch_dtype=dtype)
pipe.to(device)


video = load_video(
    "/raid/yiyi/LTX-Video/outputs/2025-03-11/video_output_0_a-woman-with-long-brown-hair-and_42_512x768x40_0.mp4"
)

condition = LTXVideoCondition(
    video=video,
    frame_index=8,
)

# Define prompts
prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt='worst quality, inconsistent motion, blurry, jittery, distorted'
# Generate the video
video = pipe(
    conditions=[condition],
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=768,
    height=512,
    num_frames=161,
    num_inference_steps=40,
).frames[0]

# Export the video
export_to_video(video, "output.mp4", fps=24)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

I don't think this is compatible with what we do in LTXRotaryPosEmbed layer... We prepare the meshgrid there and only pass the interpolation scales from the pipeline. It seems like here we are preparing the meshgrid beforehand, which will be incorrect. I think we would have to do one of the following:

• Make sure to only pass multiplicative interpolation scale without first multiplying with the latent_coords (the screenshot below shows how I handled it in the other PR)
• If we're passing latent_coords, we will have to handle it differently in the transformer for LTX v0.9.0/v0.9.1 vs v0.9.5

[yiyixuxu]

@a-r-r-o-w

I think we have different ways to handle rope in our current code base, in general, I think it's more convenient/natural to prepare position ids (e.g. the image_ids, text_ids in flux or the grid here for ltx) at same time when we patchify the latents (e.g. pack_latent for ltx or flux). flux and ltx do this in pipeline and other models like lumina handle both together inside transformer with a patch embed

I think it is ok to have this flexibility for rope since it's something that slows us down for each integration. maybe a general rule is to try to follow closer to the original code base and fit it into one of the patterns that's easier for us to maintain.

[a-r-r-o-w]

We could use float32 here and then typecast before sending into transformer, no? That way there won't be a downcast/upcast for CFG

[hlky]

Thanks @yiyixuxu

[a-r-r-o-w]

Thanks so much for taking this up @yiyixuxu! LGTM but I had one doubt

[a-r-r-o-w]

@yiyixuxu Pardon my stupidity, but I can't seem to find if we're handling this + condition.frame_index part. Is this missing by any chance, or was I mistaken in trying to handle this here?

In the original code, this is what I was meaning to handle: https://github.com/Lightricks/LTX-Video/blob/496dc5058f4408dcb777282f3fb6377fb2da08e6/ltx_video/pipelines/pipeline_ltx_video.py#L1285

[yiyixuxu]

@a-r-r-o-w it's here!