Guiding diffusion model generation using gradient similarity

[Toki912]

Here is the English translation of your summarized problem scenario:

1. An image A is passed through a ResNet network to compute its gradient ( g );
2. The intermediate image ( x_t ) during the diffusion model's generation process;
3. The gradient ( g_t ) of ( x_t ) is computed using the same ResNet network;
4. The cosine similarity between ( |g_t - g| ) is calculated as guidance information, which is used to adjust the mean and variance of the diffusion model's generation process;
5. When the diffusion model iterates to ( t = 0 ), as ( g_t ) (for ( t == 0 )) approaches ( g ), the generated image ( x_0 ) by the diffusion model also becomes highly consistent with image A.

[asomoza]

Hi, what is your question here? You're asking on how to do this? or do you have a problem when trying to implement it?

[Toki912]

Hello, thank you for your response. I am using a pre-trained DDPM
and attempting to adjust the DDPM sampling mean to generate a target image by computing the gradient of similarity.

I would like to know how to achieve this. I have tried many methods, but the issue is that the gradient similarity guidance does not seem to affect the diffusion model, in other words, the similarity guidance is not correctly steering the diffusion model’s generation.
I’m wondering if it’s possible to implement this idea using the diffusers library.

[asomoza]

oh,, sadly I can't help you with this but ccing @sayakpaul in case he has insights about your question.

[Toki912]

Thank you for your help. I wish you all the best in your work!