Genet from timm #344

Vozf · 2021-02-11T12:10:50Z

Some concerns listed in code comments.
I didn't test it on the real world task
Tests pass and weights are loading

Vozf · 2021-02-11T12:11:29Z

requirements.txt

@@ -1,4 +1,4 @@
 torchvision>=0.3.0
 pretrainedmodels==0.7.4
 efficientnet-pytorch==0.6.3
-timm==0.3.2
+git+https://github.com/rwightman/pytorch-image-models@d8e69206be253892b2956341fea09fdebfaae4e3


No release is out with genet yet, so we'll have to wait until release

Pip does not support such requirements
Possible solution: in encoders/init.py use try: except while loading this encoder with suggestion to install latest source timm (before timm new release)

Pip supports such requirement and pip install -r requirements.txt works fine. It is still a temporary solution as timm releases often and in my opinion it's easier to wait.

Vozf · 2021-02-11T12:13:07Z

segmentation_models_pytorch/encoders/timm_gernet.py

+import torch.nn as nn
+
+
+class GERNetEncoder(ByobNet, EncoderMixin):


This inherits from byobnet same as new repvgg so we should consider adding generic ByoBnetEncoder to comply with future timm changes https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/byobnet.py#L645

segmentation_models_pytorch/encoders/timm_gernet.py

Vozf · 2021-02-11T12:15:41Z

segmentation_models_pytorch/encoders/timm_gernet.py

+
+
+class GERNetEncoder(ByobNet, EncoderMixin):
+    def __init__(self, out_channels, depth=6, **kwargs):


For genet I've found 6(+rgb=7) stages and most others encoders have 5, maybe I've misunderstood the concept or we should join some stages.

each stage should reduce spatial resolution, you may combine several stages in one

Which stages should we combine if the constant resolution stages are different for each version?
(3, 13, 48, 48, 384, 560, 1920) - s
(3, 32, 128, 192, 640, 640, 2560) - m, l
Combining different stages for different versions seems odd

this is channels resolution, not spatial
try to print tensor shapes afrer each stage

suppose you have image 224 224 3
stem 112 112 13
stage_1 64 64 48
stage_2 32 32 48
stage_3 32 32 386
etc...

so, you should combine stage_2 and stage_3

s - [torch.Size([1, 3, 64, 64]), torch.Size([1, 13, 32, 32]), torch.Size([1, 48, 16, 16]), torch.Size([1, 48, 8, 8]), torch.Size([1, 384, 4, 4]), torch.Size([1, 560, 2, 2]), torch.Size([1, 1920, 2, 2])] m, l - [torch.Size([1, 3, 64, 64]), torch.Size([1, 32, 32, 32]), torch.Size([1, 128, 16, 16]), torch.Size([1, 192, 8, 8]), torch.Size([1, 640, 4, 4]), torch.Size([1, 640, 2, 2]), torch.Size([1, 2560, 2, 2])]

Last 2 should be joined as I understood?

Yes, and out_channels should be modified:
(3, 13, 48, 48, 384, 560, 1920) ->
(3, 13, 48, 48, 384, 1920)

And depth to 5

Vozf · 2021-02-11T12:18:10Z

Something wrong with cloud CI, I've checked tests locally, they pass
But if I set DEFAULT_ENCODER = "timm-gernet_s" 2 MANet tests fail

Here is the log for the failing test, 2nd one has the same
Not really sure what's going on there but it seems that Manet conv2d has 0 passed as parameter as result of skip_channels(13) // reduction(16) = 0

FAILED                            [ 71%]
tests/test_models.py:85 (test_aux_output[MAnet])
model_class = <class 'segmentation_models_pytorch.manet.model.MAnet'>

    @pytest.mark.parametrize("model_class", [smp.PAN, smp.FPN, smp.PSPNet, smp.Linknet, smp.Unet, smp.UnetPlusPlus, smp.MAnet])
    def test_aux_output(model_class):
        model = model_class(
>           DEFAULT_ENCODER, encoder_weights=None, aux_params=dict(classes=2)
        )

test_models.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../segmentation_models_pytorch/manet/model.py:78: in __init__
    pab_channels=decoder_pab_channels
../segmentation_models_pytorch/manet/decoder.py:170: in __init__
    for in_ch, skip_ch, out_ch in zip(in_channels, skip_channels, out_channels)
../segmentation_models_pytorch/manet/decoder.py:170: in <listcomp>
    for in_ch, skip_ch, out_ch in zip(in_channels, skip_channels, out_channels)
../segmentation_models_pytorch/manet/decoder.py:61: in __init__
    nn.Conv2d(skip_channels, skip_channels // reduction, 1),
../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/modules/conv.py:412: in __init__
    False, _pair(0), groups, bias, padding_mode)
../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/modules/conv.py:83: in __init__
    self.reset_parameters()
../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/modules/conv.py:86: in reset_parameters
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/init.py:379: in kaiming_uniform_
    fan = _calculate_correct_fan(tensor, mode)
../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/init.py:348: in _calculate_correct_fan
    fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

tensor = Parameter containing:
tensor([], size=(0, 13, 1, 1), requires_grad=True)

    def _calculate_fan_in_and_fan_out(tensor):
        dimensions = tensor.dim()
        if dimensions < 2:
            raise ValueError("Fan in and fan out can not be computed for tensor with fewer than 2 dimensions")
    
        num_input_fmaps = tensor.size(1)
        num_output_fmaps = tensor.size(0)
        receptive_field_size = 1
        if tensor.dim() > 2:
>           receptive_field_size = tensor[0][0].numel()
E           IndexError: index 0 is out of bounds for dimension 0 with size 0

../../../anaconda3/envs/rnd/lib/python3.6/site-packages/torch/nn/init.py:280: IndexError

Vozf · 2021-02-11T12:50:27Z

segmentation_models_pytorch/encoders/timm_gernet.py

+    for source_name, source_url in sources.items():
+        pretrained_settings[model_name][source_name] = {
+            "url": source_url,
+            'input_size': [3, 224, 224] if not model_name == 'timm-gernet_l' else [3, 256, 256],


Not sure that such if else inside builder is appropriate, but it seems better than 3 separate large configs

input_size is not necessery kwarg, you may remove it

qubvel · 2021-02-11T13:00:46Z

segmentation_models_pytorch/encoders/__init__.py

@@ -16,6 +16,7 @@
 from .timm_res2net import timm_res2net_encoders
 from .timm_regnet import timm_regnet_encoders
 from .timm_sknet import timm_sknet_encoders
+from .timm_gernet import timm_gernet_encoders


try: from .timm_gernet import timm_gernet_encoders except <Exception> as e: timm_gernet_encoders = {} <warning>

qubvel · 2021-02-11T13:11:31Z

Thanks for PR, I have left some comments about what needs to be corrected.

And also, please, add information about Encoder in following places:

README.md modify number of encoder in start description
README.md add encoder names to table
Add information about encoders to docs/encoders.rst

You can use misc/generate_table.py script to get number of params for encoders.

Vozf · 2021-02-11T13:24:10Z

Still got 2 questions regarding the Manet failing test and stage combining.

Vozf · 2021-02-11T14:23:51Z

Updated all the points. The only thing left is that manet test

qubvel · 2021-02-11T16:52:54Z

Could you patch MAnet decoder as follows and run tests:
starting from line 61:

        reduced_channels = max(1, skip_channels // reduction)
        self.SE_ll = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Conv2d(skip_channels, reduced_channels, 1),
            nn.ReLU(inplace=True),
            nn.Conv2d(reduced_channels, skip_channels, 1),
            nn.Sigmoid(),
        )
        self.SE_hl = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Conv2d(skip_channels, reduced_channels, 1),
            nn.ReLU(inplace=True),
            nn.Conv2d(reduced_channels, skip_channels, 1),
            nn.Sigmoid(),
        )

P.S. The problem is in stem, it has just 13 channels, and reducing it by 16 lead to 0 channels, which raises error.

qubvel · 2021-02-11T16:59:52Z

And please add here (after that line):
https://github.com/qubvel/segmentation_models.pytorch/blob/b07695f0e2da4626a0f69cc7672067b1c2530a6c/.github/workflows/tests.yml#L31

pip install -U git+https://github.com/rwightman/pytorch-image-models

to run tests with new encoder in CI

Vozf · 2021-02-11T17:14:59Z

All passing now

Vozf · 2021-02-12T06:58:02Z

Set training genet_s model for the night, and its val metrics are noticeably worse than efficientnet-b0(not from timm, I've faced some similar val metric issues with timm previously)
No clue why is this happening and how much expected is such behavior

qubvel · 2021-02-15T08:30:40Z

I will try to test it on some of my tasks, and then merge

Vozf · 2021-03-17T10:53:23Z

timm is released

Vozf · 2021-07-04T16:08:55Z

pip install ... should probably be removed from tests as timm is already released

AlexanderYaroshevichIAC added 2 commits February 11, 2021 13:18

gernet from regnet

3511221

basic gernet

099c08f

Vozf commented Feb 11, 2021

View reviewed changes

qubvel reviewed Feb 11, 2021

View reviewed changes

AlexanderYaroshevichIAC added 4 commits February 11, 2021 17:07

depth set to 5, and requirements+import update

cc42d1c

docs

bc92a9f

Fix summary error

3b1961b

remove input size

97f8114

manet fix and test with latest timm

e5e19dc

Merge branch 'master' into master

0d8c7a6

qubvel merged commit 23a54b4 into qubvel-org:master Jul 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Genet from timm #344

Genet from timm #344

Vozf commented Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021

qubvel Feb 11, 2021

Vozf Feb 11, 2021

Vozf Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021 •

edited

Loading

qubvel Feb 11, 2021

Vozf Feb 11, 2021 •

edited

Loading

qubvel Feb 11, 2021

qubvel Feb 11, 2021

Vozf Feb 11, 2021

qubvel Feb 11, 2021

qubvel Feb 11, 2021

Vozf commented Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021

qubvel Feb 11, 2021

qubvel Feb 11, 2021

qubvel commented Feb 11, 2021

Vozf commented Feb 11, 2021

Vozf commented Feb 11, 2021

qubvel commented Feb 11, 2021 •

edited

Loading

qubvel commented Feb 11, 2021

Vozf commented Feb 11, 2021

Vozf commented Feb 12, 2021 •

edited

Loading

qubvel commented Feb 15, 2021

Vozf commented Mar 17, 2021

Vozf commented Jul 4, 2021

		import torch.nn as nn


		class GERNetEncoder(ByobNet, EncoderMixin):



		class GERNetEncoder(ByobNet, EncoderMixin):
		def __init__(self, out_channels, depth=6, **kwargs):

Genet from timm #344

Genet from timm #344

Conversation

Vozf commented Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vozf Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Vozf Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vozf Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vozf commented Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qubvel commented Feb 11, 2021

Vozf commented Feb 11, 2021

Vozf commented Feb 11, 2021

qubvel commented Feb 11, 2021 • edited Loading

qubvel commented Feb 11, 2021

Vozf commented Feb 11, 2021

Vozf commented Feb 12, 2021 • edited Loading

qubvel commented Feb 15, 2021

Vozf commented Mar 17, 2021

Vozf commented Jul 4, 2021

Vozf commented Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021 •

edited

Loading

Vozf Feb 11, 2021 •

edited

Loading

Vozf commented Feb 11, 2021 •

edited

Loading

qubvel commented Feb 11, 2021 •

edited

Loading

Vozf commented Feb 12, 2021 •

edited

Loading