DiffusionDet: Diffusion models for object detection

[345ishaan]

Model/Pipeline/Scheduler description

Recent work which leverages diffusion models for object detection task. https://arxiv.org/abs/2211.09788

Add capability to run it through HF diffusers pipeline and if possible also create benchmarks or comparison on datasets like nuScenes.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[patrickvonplaten]

Model weights seem to be available as well no? https://github.com/ShoufaChen/DiffusionDet#models

[vvvm23]

@patrickvonplaten mind if I give this a try? This would be my first time contributing a model, so I might need a hand occasionally.

[345ishaan]

@patrickvonplaten mind if I give this a try? This would be my first time contributing a model, so I might need a hand occasionally.

Sorry I forgot to assign the issue to myself while opening but was actually planning to look into this.

[vvvm23]

Go for it! 🤗

[345ishaan]

Go for it! 🤗

I am happy to collaborate if you want :) I have done it in the past and given I will be only working outside office-hours, things can move faster that way.

[vvvm23]

Ordinarily I would say yes, but I don't think I can dedicate any time towards it until Tuesday at the earliest. So probably best for you to make a start yourself and if you have anything you want to hand off to me, I can chip in a bit 😅

[patrickvonplaten]

Also more than happy to help if needed :-)

[345ishaan]

@patrickvonplaten Plan to run through their code this weekend in inference mode. Let me know if you have a task checklist in your mind which i should be following. Happy to split up if needed.

[patrickvonplaten]

Sure, I think the following would make sense:

1. Get the pipeline working witht the original codebase
2. Add the core unet model to diffusers
   a) First make sure weights can be correctly
   b) Then check forward pass
3. Add remaining components

Also happy to guide you through a PR :-)

[345ishaan]

I am able to run the original codebase here: https://colab.research.google.com/drive/1rA5SXuTx2pI6o7tWA6Ad5QRZn4a1ajMh#scrollTo=Sn5gWF3fhpf-

[345ishaan]

@patrickvonplaten This work is tried on types of encoder, CNN based (Resnet Style) and Transformer based (Swin Transformer). Do you prefer transformer based, also is there a HF implementation of Swin Transformer which i should refer?

[patrickvonplaten]

Think leveraging an existing transformers implementation could make a lot of sense here (also cc @ShoufaChen as the author of diffusionDet :-) )

And maybe @NielsRogge FYI

[NielsRogge]

Yes we do have Swin implemented here -> https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin/modeling_swin.py. So you can do from transformers import SwinModel