Running `diffusers` with `torch`-DDP leads to warning on stride mismatch

[eeshandhekane]

Describe the bug

Hey folks!
First off, thanks a lot for the amazing work with diffusers!
Recently, we have observed a warning while using diffusers-based models and running scripts.
We also observe similar warning while using only the models provided in diffusers/models/.
While this error might not be critical, we wanted to highlight the issue.
Thanks in advance!

Reproduction

Below are the steps that can be taken to observe this warning:

1. Download the diffusers repository as mentioned in the documentation.
2. Set a run with train_unconditional.py script as mentioned in the Unconditional image generation tutorial.
3. Use vanilla torch DDP.

With this setup, we observe that we hit a warning about stride mismatch in the gradients and the bucket view:

Logs

No response

System Info

We are using the following version of different packages:
Diffusers: diffusers==0.15.0
Torch: torch==1.13.0
Python: Python 3.9.16

Who can help?

No response

[patrickvonplaten]

Interesting! Thanks for the heads-up, gently pinging @williamberman and @sayakpaul here

[williamberman]

Neat @eeshandhekane ! afaik accelerate is what handles manually wrapping in ddp under the hood, I'm not familiar with this warning and what the performance implications are. Might be worth starting a thread on the accelerate forum. W/out concrete numbers on what the performance impacts are I'm not sure how actionable it is for us :)

[sayakpaul]

Would also suggest trying to replicate this in a minimal manner (potentially with a similar but smaller model).

I am going to close this thread for now, as there's nothing actionable here for us as of yet. As @williamberman suggested, maybe open a thread here: https://github.com/huggingface/accelerate.