[Tracker] use micro-conditioning for the SDXL trainers

[sayakpaul]

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper here.

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

• DreamBooth SDXL LoRA
• SDXL LCM-LoRA (it needs to be slightly refactored as done in Fix a bug of flip in SDXL training script #6547 by @haofanwang)
• ControlNet SDXL
• T2I Adapter SDXL
• Textual inversion SDXL (same as what's mentioned for SDXL LCM-LoRA above)
• Advanced SDXL trainer

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

[sayakpaul]

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

[charchit7]

Hey @sayakpaul what's the min requirement for GPU while testing for the training?

[sayakpaul]

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

[linoytsaban]

@linoytsaban would you like to take Advanced SDXL trainer?

sure!

[jiqing-feng]

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

Sorry for the delay, I have been quite busy recently; I will do it once I have time.

[ChengsongLu]

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

Can it run locally?

[sayakpaul]

You need to repurpose the code accordingly for that.

[cosmo3769]

Hello @sayakpaul, I would love to work on ControlNet SDXL. Thank you.

[sayakpaul]

Please go ahead.

[jiqing-feng]

Hi @sayakpaul . I think the micro conditioning is already in the example, see conditioning_image_size and conditioning_crop_size, and they are all in the add_time_ids.

BTW, I found a minor error about the resize function, and I already fixed it in #7095

[sayakpaul]

You are right. Thanks for the quick fix.