[BUG] out_indices are not always checked

[f-fuchs]

Describe the bug
For some models, it is possible to specify out_indices that are not viable. Instead of failing, the create_model function returns a model with the non-viable indices missing.

To Reproduce
Steps to reproduce the behavior:

import torch
from timm import create_model

model = create_model(
    "deit3_base_patch16_224.fb_in22k_ft_in1k",
    pretrained=True,
    exportable=True,
    in_chans=1,
    out_indices=[3, 5, 7, 100],
    features_only=True,
)

shape = (1, 1, 224, 224)
input_ = torch.randn(shape)

output_ = model(input_)

for feature in output_:
    print(f"{feature.shape=}")

feature.shape=torch.Size([1, 768, 14, 14])
feature.shape=torch.Size([1, 768, 14, 14])
feature.shape=torch.Size([1, 768, 14, 14])

Expected behavior
If not an error, at least a warning that some indices that where specified will be ignored.

Desktop (please complete the following information):

• Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
• Python version: 3.10.14
• PyTorch version (GPU?): 2.3.1+cu121 (False)
• timm version: 1.0.7

Incorrectly working models

• deit3_base_patch16_224.fb_in22k_ft_in1k
• eva02_base_patch14_224.mim_in22k

Correctly? working models

• tf_efficientnet_b7.ns_jft_in1k

Traceback (most recent call last):
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/minimal_example.py", line 4, in <module>
    model = create_model(
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/_factory.py", line 117, in create_model
    model = create_fn(
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/efficientnet.py", line 2307, in tf_efficientnet_b7
    model = _gen_efficientnet(
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/efficientnet.py", line 651, in _gen_efficientnet
    model = _create_effnet(variant, pretrained, **model_kwargs)
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/efficientnet.py", line 368, in _create_effnet
    model = build_model_with_cfg(
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/_builder.py", line 406, in build_model_with_cfg
    model = model_cls(**kwargs)
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/efficientnet.py", line 321, in __init__
    self._stage_out_idx = {f['stage']: f['index'] for f in self.feature_info.get_dicts()}
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/_features.py", line 120, in get_dicts
    return [self.info[i] for i in self.out_indices]
  File "/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/timm/models/_features.py", line 120, in <listcomp>
    return [self.info[i] for i in self.out_indices]
IndexError: list index out of range

Not sure if this is by design or accident, but EfficentNet throws an error.

[rwightman]

@f-fuchs not by design, just didn't add the extra checks, default mechanism for vit is quite different than efficientnet

[fewroad]

I want to use the SwinTransformer and I also meet the bug, what should I do?

[rwightman]

@fewroad I don't believe you did, swin's default features_only=True mechanism will behave like the mentioned efficientnet

mm = timm.create_model('swin_tiny_patch4_window7_224', pretrained=True, features_only=True, out_indices=(0, 100)) yields

    return [self.info[i][key] for i in self.out_indices]
            ~~~~~~~~~^^^
IndexError: list index out of range

If you're using the forward_intermediates fn then it will have this issue, but that's been fixed by the merged commit.

[f-fuchs]

thx for the quick fix 👍 @rwightman

@fewroad you should try huggingfaces transformers library for SwinTransformer, because atleast when I tried the SwinTransformer from timm I did not support variable input sizes for inference.

[fewroad]

@rwightman Sorry, I think the "Indexerror" is the bug, I try to correct the error but I don't know how to fix it.
@f-fuchs Thank you for your response, but what is the reason for the "Indexerror", I don't overcome it. Hope for your reply!

[f-fuchs]

@fewroad what indices are you trying?

this works for me.

import torch
from timm import create_model

model = create_model(
    "swin_tiny_patch4_window7_224",
    pretrained=True,
    exportable=True,
    in_chans=1,
    out_indices=[0, 1, 2, 3],
    features_only=True,
)

shape = (1, 1, 224, 224)
input_ = torch.randn(shape)
output = model(input_)
for feature in output:
    print(f"{feature.shape=}")

feature.shape=torch.Size([1, 56, 56, 96])
feature.shape=torch.Size([1, 28, 28, 192])
feature.shape=torch.Size([1, 14, 14, 384])
feature.shape=torch.Size([1, 7, 7, 768])

[rwightman]

@f-fuchs timm swin and swin v2 supports resizing, but it's not done on the fly, to do it properly the window size should be rescaled with the img size, etc and I don't believe the transformers always window size change on load or on the fly? Maybe that was added....

Can also use non-square windows and image sizes in timm.

model = create_model(
    "swin_tiny_patch4_window7_224",
    pretrained=True,
    exportable=True,
    in_chans=1,
    out_indices=[0, 1, 2, 3],
    features_only=True,
    img_size=(256, 384),
    window_size=(8, 12),
)

The branch #2225 will add support for a set_input_size() method that can be called after model creation, but still not in forward as it requires rebuilding the pairwise pos encoding which is expensive to do in forward, can also resize the patch size.

EDIT: eval result using that 256x384 image size above with 8x12 window ... so doing the proper window resize we get 80.5 eval result, using the transformers yolo approach and permitting any resolution without resizing window yields 78.9.

I can easily disable the resolution / patch grid check to allow that by default, but it's not really the correct approach in many situation.

[rwightman]

@fewroad the out_indices for using features_only=True using default feat extractor settins match the feature_info in the model, you can't use indices beyond that. The index error bug means you're asking for a feature that isn't available.

Also, FYI you can use -ve indexing. So -1 is the last feature, -2 the penultimate, etc

[rwightman]

since I'm working on the set_input_size fn right now and it touches all this, I'm adding a strict_img_size argument to init ... if set to False on model creation it'll allow any img size without complaining or resizing window but as mentioned, results are less than optimal in many cases.

[rwightman]

you can try it out on branch for #2225 now, currently running more tests...

if you pass strict_img_size=False to any of the swin v1/v2/v2-cr it will behave like transformers, it won't check the image size, create mask on the fly in forward, leave window size as is. Not always optimal but quick and easy.

always_partition was also added before this last change and can be enabled to force the full window size + shift even when the resolution is < window

passing img_size= and/or window_size= to on model create will resize the pretrained weights at load time

calling set_input_size() once model is created, with img_size, window_size, patch_size (patch resize not for v2_cr though) will resize any/all of those. If window size is left None the window will be resized to keep the ratio.

[f-fuchs]

hey thanks for your fast response and explanations. appreciate the help.

I have one question though. does the performance decrease also also affect image sizes that are dividable by window size times patch size?