[Q] Are there any ViT models available that were finetuned on COCO?

[NightMachinery]

Are there any ViT models available that were finetuned on COCO?

In particular, I am looking for a ViT that can classify COCO. (I want to benchmark a weaky supervised segmentation technique using such a model.)

I see that open_clip has ('coca_ViT-B-32', 'mscoco_finetuned_laion2b_s13b_b90k'), but I assume that is only useful for captioning and not classification?

[NightMachinery]

I am also interested in any models at all that can do classification on COCO.

[rwightman]

@NightMachinery nothing I'm aware of, pure classification models train on COCO are generally pretty uncommon

[NightMachinery]

I aim to fine-tune a model, specifically vit_base_patch16_224.augreg2_in21k_ft_in1k, for multi-label classification on the COCO dataset. My plan is to extract logits from the classification head (this part doesn't need any code changes to the ViT model provided in timm), apply sigmoid to each, and then calculate the mean of binary cross entropy loss computed for each class, as discussed here. I'm considering whether this is an effective strategy. What is the most appropriate learning rate scheduler to use in this context?

Additionally, I'm seeking resources or guides that could be beneficial to review before starting this process. For instance, would the paper "How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers" provide valuable insights? Furthermore, are there any specific tutorials on fine-tuning a model with timm that I should look into?

Should I use data augmentations? Are these augmentations available in timm already?