GeneralizedRCNNTransform output different shapes of images and targets

[lpq97lpq]

🐛 Describe the bug

I would expect the shape of images and targets consistent with each other (that is, have the same width and height).

However, since torchvision.models.detection.transform.GeneralizedRCNNTransform call self.batch_images at the end of its forward function only on images, the output images and targets usually don't have the same shapes.

Example:

from torchvision.models.detection.transform import GeneralizedRCNNTransform
import torch

#initialize transformation
image_mean = [0.485, 0.456, 0.406]
image_std = [0.229, 0.224, 0.225]
T=GeneralizedRCNNTransform(800,1333,image_mean,image_std)

#initialize image & target
images=[torch.randn([3,1000,900])]
targets=[{'boxes':torch.tensor([[0,0,1000,900]]),'masks':torch.randn([1,1000,900]).byte()}]

#do experiment
images,targets=T(images,targets)

print(images.tensors.shape) 
print(images.image_sizes)
print(targets[0]['boxes'])
print(targets[0]['masks'].shape)

output:

torch.Size([1, 3, 896, 800])
[(888, 800)]
tensor([[ 0.0000, 0.0000, 888.8889, 799.2000]])
torch.Size([1, 888, 800])

The default size_divisible parameter in self.batch_images is 32. Since 888 is not divisible by 32, it pads the image (888,800) to
(896,800). But there is no change on 'boxes' and 'masks' and even images.image_sizes.

Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.8 (default, Apr 13 2021, 19:58:26) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.11.0-37-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.4.120
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070
Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.1
[pip3] numpydoc==1.1.0
[pip3] pytorch-lightning==1.3.3
[pip3] torch==1.11.0
[pip3] torch-summary==1.4.5
[pip3] torchaudio==0.11.0
[pip3] torchmetrics==0.5.1
[pip3] torchvision==0.12.0
[pip3] vit-pytorch==0.22.0
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.2.0 h06a4308_296
[conda] mkl-service 2.3.0 py38h27cfd23_1
[conda] mkl_fft 1.3.0 py38h42c9631_2
[conda] mkl_random 1.2.1 py38ha9443f7_2
[conda] numpy 1.20.1 py38h93e21f0_0
[conda] numpy-base 1.20.1 py38h7d8b39e_0
[conda] numpydoc 1.1.0 pyhd3eb1b0_1
[conda] pytorch 1.11.0 py3.8_cuda11.3_cudnn8.2.0_0 pytorch
[conda] pytorch-lightning 1.3.3 pypi_0 pypi
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 1.11.0 pypi_0 pypi
[conda] torch-summary 1.4.5 pypi_0 pypi
[conda] torchaudio 0.11.0 py38_cu113 pytorch
[conda] torchmetrics 0.5.1 pypi_0 pypi
[conda] torchvision 0.12.0 pypi_0 pypi
[conda] vit-pytorch 0.22.0 pypi_0 pypi

cc @datumbox @vfdev-5 @YosuaMichael

[datumbox]

@lpq97lpq Thanks for raising this. I agree, this is very confusing.

The GeneralizedRCNNTransform is not meant to be used directly by users (though that's not really obvious). In the current Detection API (which is still Beta), the models include the transforms internally. This was done so that they can undo internal resizing ops such as the ones you point out. Here is the intended usage:

import torch

images=[torch.randn([3,1000,900])]
from torchvision.models.detection import maskrcnn_resnet50_fpn_v2

model = maskrcnn_resnet50_fpn_v2().eval()
predictions = model(images)


print(images[0].shape)
print(predictions[0]["masks"].shape)

The above prints:

torch.Size([3, 1000, 900])
torch.Size([100, 1, 1000, 900])

This means that the resizing done by size_divisible is not visible to the users.

The API of Detection models has multiple pros/cons. Here are the pros:

• The model can do all sorts of internal optimizations and these are transparent to the user
• The model is capable to perform internally postprocessing (such as NMS) and resizing and return back to the user the expected results.
• The API can support 2-stage models where different components have different losses without the user needing to worry about them.

But there are also cons:

• The detection models look nothing like the standard Model paradigm in PyTorch. The losses are parts of the model, the forward behaves differently depending on whether we do train/eval, the images are transformed within the model instead of outside.
• There is limitation on the amount of data augmentation strategies we can do. Overtime we had to introduce hacky and ugly patches to make the use of modern augmentations possible (see this workaround for which I'm not too proud: https://github.com/pytorch/vision/pull/5715/files#r839744536)

The Detection API is in need of a revamp. It will require careful thinking on how this will be done because one can't optimize for all use-cases. One direction we consider taking is to make detection models compatible with the standard paradigm of Models in PyTorch. That would include moving Transforms and Losses out of the model and requiring the user to explicitly do the conversions themselves. Unfortunately this would also mean handling 2-stage models explicitly by calling their components and losses directly; that is far from ideal. We haven't finalized the plan on this and currently we are running low on headcount to prioritize this work. But I will leave your issue open as a great example of the things we need to improve concerning the API. Thank you for taking the time to report the issue. Let me know your thoughts.

[lpq97lpq]

Thank you for your detailed explanation.

But correct me if I am wrong, will this inconstancy affects the training and loss function? In the GeneralizedRCNNTransform function, the feature map is directly calculated by

vision/torchvision/models/detection/generalized_rcnn.py

Line 101 in 7ba9719

features = self.backbone(images.tensors)

Thus, with or without self.batch_images, the features map will be different not only in value but also in shape. However there is no way the Rpn and Roi (loss) function can tell the difference since they can only see the same ground truth.

vision/torchvision/models/detection/generalized_rcnn.py

Lines 104 to 105 in 7ba9719

	proposals, proposal_losses = self.rpn(images, features, targets)
	detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)

(for Rpn part, the anchor_generator do re-calculate the real image size but the loss do not )

vision/torchvision/models/detection/rpn.py

Line 365 in 7ba9719

anchors = self.anchor_generator(images, features)

vision/torchvision/models/detection/rpn.py

Lines 376 to 390 in 7ba9719

	boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
	losses = {}
	if self.training:
	if targets is None:
	raise ValueError("targets should not be None")
	labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
	regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
	loss_objectness, loss_rpn_box_reg = self.compute_loss(
	objectness, pred_bbox_deltas, labels, regression_targets
	)
	losses = {
	"loss_objectness": loss_objectness,
	"loss_rpn_box_reg": loss_rpn_box_reg,
	}

I think this behavior will affect the final accuracy of neural network.
Thank you very much.

[datumbox]

@lpq97lpq Thanks for insisting on getting clarifications. It's important that we get on the bottom of the problem you are raising.

You are right to say that the transforms leave momentarily the images and the targets desynced in terms of size on the following all:

vision/torchvision/models/detection/generalized_rcnn.py

Line 83 in 7ba9719

images, targets = self.transform(images, targets)

This is because the forward of the transform uses size_divisible to resize only the images, not the targets:

vision/torchvision/models/detection/transform.py

Lines 135 to 136 in 7ba9719

	image_sizes = [img.shape[-2:] for img in images]
	images = self.batch_images(images, size_divisible=self.size_divisible)

But as you see before doing the batching, we grab the sizes of the images. This information is later provided to RoIHead:

vision/torchvision/models/detection/generalized_rcnn.py

Line 105 in 7ba9719

detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)

The forward of RoIHeads factors in the original image sizes in the box_roi_pool estimations. Unfortunately, it is not clear to me if this is where the alignment between images and targets is achieved. @fmassa would you be able to provide some insights on how this is achieved?

[lpq97lpq]

Thank you for your kindly help.

Actually I miss an important point. Since the

images = self.batch_images(images, size_divisible=self.size_divisible) 

is doing padding, It won't affect the boxes and masks a lot: the boxes is unchanged and we just need to do the same padding operation on masks.

Thus, it should be ok to leave targets unchanged, and the loss won't differ a lot.

There is another example:

import torch
from torchvision.models.detection import maskrcnn_resnet50_fpn_v2
import copy
from torch.nn import functional as F

SEED = 100
torch.manual_seed(SEED)

#initialize image & target
images=[torch.randn([3,1000,900])]
targets=[{'boxes':torch.tensor([[0,0,1000,900],[0,0,1000,900],[0,0,1000,900]]),'labels':torch.tensor([1,2,1]),'masks':(torch.randn([3,1000,900])>0).byte()}]
model = maskrcnn_resnet50_fpn_v2()
images, targets = model.transform(images, targets)

with torch.no_grad():
    features = model.backbone(images.tensors)

    #calculate the original loss
    torch.manual_seed(SEED)
    loss_origin={}
    proposals, proposal_losses = model.rpn(images, features, targets)
    detections, detector_losses = model.roi_heads(features, proposals, images.image_sizes, targets)
    loss_origin.update(proposal_losses)
    loss_origin.update(detector_losses)
    

    #calculate the adjust loss
    images_adjust=copy.deepcopy(images) 
    targets_adjust=copy.deepcopy(targets)
    pad=[0,0,0,0]
    pad[1]=images_adjust.tensors.size(-1)-images_adjust.image_sizes[0][-1]
    pad[3]=images_adjust.tensors.size(-2)-images_adjust.image_sizes[0][-2]
    targets_adjust[0]['masks']=F.pad(targets_adjust[0]['masks'], pad)
    images_adjust.image_sizes=[(images_adjust.tensors.size(-2),images_adjust.tensors.size(-1))] #you can commnet this line to get closer loss output
    
    torch.manual_seed(SEED)
    loss_adjust={}
    proposals, proposal_losses = model.rpn(images_adjust, features, targets_adjust)
    detections, detector_losses = model.roi_heads(features, proposals, images_adjust.image_sizes, targets_adjust)
    loss_adjust.update(proposal_losses)
    loss_adjust.update(detector_losses)
    
    
loss_differ={}
for key in loss_origin:
    loss_differ[key]=loss_origin[key]-loss_adjust[key]
print(loss_differ)
    
    
    
#do more padding
pad=[0,1000,0,1000]
images.tensors=F.pad(images.tensors, pad)

with torch.no_grad():
    features = model.backbone(images.tensors)
    
    #calculate the original loss
    torch.manual_seed(SEED)
    loss_origin={}
    proposals, proposal_losses = model.rpn(images, features, targets)
    detections, detector_losses = model.roi_heads(features, proposals, images.image_sizes, targets)
    loss_origin.update(proposal_losses)
    loss_origin.update(detector_losses)
    
    
    #calculate the adjust loss
    images_adjust=copy.deepcopy(images)
    targets_adjust=copy.deepcopy(targets)
    pad=[0,0,0,0]
    pad[1]=images_adjust.tensors.size(-1)-images_adjust.image_sizes[0][-1]
    pad[3]=images_adjust.tensors.size(-2)-images_adjust.image_sizes[0][-2]
    targets_adjust[0]['masks']=F.pad(targets_adjust[0]['masks'], pad)
    images_adjust.image_sizes=[(images_adjust.tensors.size(-2),images_adjust.tensors.size(-1))]  #you can commnet this line to get closer loss output
    
    torch.manual_seed(SEED)
    loss_adjust={}
    proposals, proposal_losses = model.rpn(images_adjust, features, targets_adjust)
    detections, detector_losses = model.roi_heads(features, proposals, images_adjust.image_sizes, targets_adjust)
    loss_adjust.update(proposal_losses)
    loss_adjust.update(detector_losses)
    
    
loss_differ={}
for key in loss_origin:
    loss_differ[key]=loss_origin[key]-loss_adjust[key]
print(loss_differ)

comment the line

 images_adjust.image_sizes=[(images_adjust.tensors.size(-2),images_adjust.tensors.size(-1))] #you can commnet this line to get closer loss output

will get closer loss output, which indicates that the GeneralizedRCNNTransform has some affect on the training and loss function.

output with perfect alignment:

{'loss_objectness': tensor(0.), 'loss_rpn_box_reg': tensor(0.), 'loss_classifier': tensor(-0.0041), 'loss_box_reg': tensor(9.2079e-05), 'loss_mask': tensor(0.)}
{'loss_objectness': tensor(0.), 'loss_rpn_box_reg': tensor(0.), 'loss_classifier': tensor(-0.0005), 'loss_box_reg': tensor(9.2634e-05), 'loss_mask': tensor(4.2975e-05)}

output with above line commented:

{'loss_objectness': tensor(0.), 'loss_rpn_box_reg': tensor(0.), 'loss_classifier': tensor(0.), 'loss_box_reg': tensor(0.), 'loss_mask': tensor(0.)}
{'loss_objectness': tensor(0.), 'loss_rpn_box_reg': tensor(0.), 'loss_classifier': tensor(0.), 'loss_box_reg': tensor(0.), 'loss_mask': tensor(4.2975e-05)}

[fmassa]

Hi,

Thanks for raising this question. There is a slight misunderstanding in one point in here though.

batch_images adds padding on the bottom-right side of the images (if needed). Note that the bounding boxes are referenced starting from the top-left corner of the image, in absolute coordinates.

This means that indeed performing the batching as it was implemented doesn't affect the coordinates of the boxes.

Note that this wouldn't hold true if the boxes were represented as relative coordinates (i.e., in 0-1), because the padding done in the images would affect the value required in the boxes for them to represent the same value (e.g., 1.0 in relative coordinates for an image of size 100, if padded to size 200, would correspond to 0.5).

The slight difference in values of doing padding is due to the convolutions in the backbone having slightly different results due to padding, but this is very small as a difference and shouldn't matter much in practice

I hope this answers the question.