Wrong box3d_overlap output

[mikeedi]

Strange behaviour of pytorch3d.ops.box3d_overlap

First case - zero overlap expected, but non-zero value is returned (and also different values depending on the order of the arguments):

import torch
from pytorch3d.ops import box3d_overlap
a = torch.tensor([[-1.0000, -1.0000, -0.5000],
        [ 1.0000, -1.0000, -0.5000],
        [ 1.0000,  1.0000, -0.5000],
        [-1.0000,  1.0000, -0.5000],
        [-1.0000, -1.0000,  0.5000],
        [ 1.0000, -1.0000,  0.5000],
        [ 1.0000,  1.0000,  0.5000],
        [-1.0000,  1.0000,  0.5000]])
b = torch.tensor([[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])

print(box3d_overlap(a[None], b[None]))
print(box3d_overlap(b[None], a[None]))

Returns:

(tensor([[4.0000]]), tensor([[inf]]))
(tensor([[0.]]), tensor([[0.]]))

Second case:

import torch
from pytorch3d.ops import box3d_overlap
# function from https://github.com/facebookresearch/pytorch3d/blob/main/tests/test_iou_box3d.py 
def create_box(xyz, whl):
    x, y, z = xyz
    w, h, le = whl

    verts = torch.tensor(
        [
            [x - w / 2.0, y - h / 2.0, z - le / 2.0],
            [x + w / 2.0, y - h / 2.0, z - le / 2.0],
            [x + w / 2.0, y + h / 2.0, z - le / 2.0],
            [x - w / 2.0, y + h / 2.0, z - le / 2.0],
            [x - w / 2.0, y - h / 2.0, z + le / 2.0],
            [x + w / 2.0, y - h / 2.0, z + le / 2.0],
            [x + w / 2.0, y + h / 2.0, z + le / 2.0],
            [x - w / 2.0, y + h / 2.0, z + le / 2.0],
        ],
        device=xyz.device,
        dtype=torch.float32,
    )
    return verts


ctrs = torch.tensor([[0., 0., 0.], [-1., 1., 0.]])
whl = torch.tensor([[2., 2., 2.], [2., 2, 2]])
box8a = create_box(ctrs[0], whl[0])
box8b = create_box(ctrs[1], whl[1])
print(box3d_overlap(box8a[None], box8b[None]))

Returns:

(tensor([[1.3704]]), tensor([[0.0937]]))

if you calculate the overlap area by hand, you get exactly 2, but the function gives a different result.

Moreover, if you slightly change the values, then the answer will be correct:

ctrs = torch.tensor([[0., 0., 0.], [-0.999999, 1., 0.]])
whl = torch.tensor([[2., 2., 2.], [2., 2, 2]])
box8a = create_box(ctrs[0], whl[0])
box8b = create_box(ctrs[1], whl[1])
print(box3d_overlap(box8a[None], box8b[None]))

ctrs = torch.tensor([[0., 0., 0.], [-1., 1., 0.]])
whl = torch.tensor([[2., 2., 2.], [2., 2, 2]])
box8a = create_box(ctrs[0], whl[0])
box8b = create_box(ctrs[1], whl[1])
print(box3d_overlap(box8a[None], box8b[None]))

ctrs = torch.tensor([[0., 0., 0.], [-1.000001, 1., 0.]])
whl = torch.tensor([[2., 2., 2.], [2., 2, 2]])
box8a = create_box(ctrs[0], whl[0])
box8b = create_box(ctrs[1], whl[1])
print(box3d_overlap(box8a[None], box8b[None]))

Returns:

(tensor([[2.0002]]), tensor([[0.1429]]))
(tensor([[1.3704]]), tensor([[0.0937]]))
(tensor([[1.9998]]), tensor([[0.1428]]))

Was checked with

torch == 1.6.0+cu101
pytorch3d==0.6.1

[nikhilaravi]

@mikeedi thanks for raising this issue. I think this might be due to the setting of an epsilon value. We will look into it!

[gkioxari]

Thank you for bringing this up!

In the first case, the second box has all its vertices at (0,0,0) which is an invalid box. We should likely check for this as the algorithm will not handle this gracefully. We check that boxes have coplanar points in the sides, but we should check that the area of the planes is nonzero.

The second case should work and it's a mystery why it doesn't. Our tests have similar box configurations. We will look into that.

[lysukhin]

Hi, @nikhilaravi @gkioxari !

I got some confusing results to report as well concerning box 3d overlap.
I wrote some custom tests and found out that iou3d of a unit box at the origin with itself is not always equal to 1 (not even close to it sometimes).

Even more confusing is the fact that the chance of a 'wrong' self-IoU is different for CPU (about 5% of the time, see code below) and GPU (about 30%).

Code for reproducing:

import torch
import numpy as np
import tqdm

from pytorch3d.ops import box3d_overlap


num_random_runs = 10000
max_diff = 1e-2

# device = "cpu"
device = "cuda:0"


def rotation_matrix_around_y(yaw):
    cos_yaw = np.cos(yaw)
    sin_yaw = np.sin(yaw)
    return np.asarray([
        [cos_yaw,   0,  sin_yaw],
        [0,         1,        0],
        [-sin_yaw,  0,  cos_yaw]
    ])


def main():

    x = y = z = 0.
    l = w = h = 1.

    x_corners = np.asarray([-1, 1, 1, -1, -1, 1, 1, -1], dtype=np.float32) * l / 2
    y_corners = np.asarray([-1, -1, 1, 1, -1, -1, 1, 1], dtype=np.float32) * h / 2
    z_corners = np.asarray([-1, -1, -1, -1, 1, 1, 1, 1], dtype=np.float32) * w / 2
    _corners = np.stack((x_corners, y_corners, z_corners), axis=1)  # -> 8 x 3
    shift = np.asarray([x, y, z], dtype=np.float32)[None, :]

    ious = []
    yaws = []
    for i in tqdm.trange(num_random_runs):
        yaw = np.random.uniform(-np.pi, np.pi)

        corners = (rotation_matrix_around_y(yaw) @ _corners.T).T
        corners += shift
        corners = torch.from_numpy(corners).float().unsqueeze(0).to(device)

        vol, iou = box3d_overlap(corners, corners)
        ious.append(iou.item())
        yaws.append(yaw)

    correct = 0
    for yaw, iou in zip(yaws, ious):

        if abs(iou - 1.) < max_diff:
            correct += 1
        # else:
        #     print(yaw, iou)

    print("% of correct self-IoU3d:", correct / num_random_runs)


if __name__ == "__main__":
    main()

[gkioxari]

@mikeedi
This is now fixed with ccfb72c

For the first case, we check to see if the sides of the input boxes have non zero area.

The second case was a tricky one and it boiled down to a numerical issue which is fixed by a small change in the algorithm. It's a nuance so I won't get into too much detail but it resulted from the fact that your two input boxes had coplanar sides. This is why when you slightly perturbed the center of the box, the output was correct. I didn't catch this case when I implemented this so thank you very much for bringing this up and giving us reproducible code. This is precisely what I enjoy about github communities! Let us know if you have any more issues with it!

[gkioxari]

@lysukhin
I ran your code and I found it to always be 1.0 for all runs, for cuda and cpu. What version are you using?I am on 0.6.1

def rotation_matrix_around_y(yaw):
    cos_yaw = np.cos(yaw)
    sin_yaw = np.sin(yaw)
    return np.asarray([[cos_yaw, 0, sin_yaw], [0, 1, 0], [-sin_yaw, 0, cos_yaw]])


num_random_runs = 10000
num_false = 0

x = y = z = 0.0
l = w = h = 1.0

device = torch.device("cuda:0")
x_corners = np.asarray([-1, 1, 1, -1, -1, 1, 1, -1], dtype=np.float32) * l / 2
y_corners = np.asarray([-1, -1, 1, 1, -1, -1, 1, 1], dtype=np.float32) * h / 2
z_corners = np.asarray([-1, -1, -1, -1, 1, 1, 1, 1], dtype=np.float32) * w / 2
_corners = np.stack((x_corners, y_corners, z_corners), axis=1)  # -> 8 x 3
shift = np.asarray([x, y, z], dtype=np.float32)[None, :]

for i in range(num_random_runs):
    yaw = np.random.uniform(-np.pi, np.pi)

    corners = (rotation_matrix_around_y(yaw) @ _corners.T).T
    corners += shift
    corners = torch.from_numpy(corners).float().unsqueeze(0).to(device)

    vol, iou = box3d_overlap(corners, corners)
    if torch.abs(iou - 1.0) > 1e-2:
        num_false += 1

print("[False] %.2f %%" % (num_false / num_random_runs * 100.0))

>> [False] 0.00 %

[lysukhin]

@gkioxari
Thanks for your answer, but that's interesting:
I re-ran your version of code and got [False] 6.09% for cpu and [False] 32.16% for cuda:0.
Moreover, I frequently face ValueError: Plane vertices are not coplanar while using cuda:0 (and never for cpu in my experiments).

Output of pytorch3d.__version__ is 0.6.0.

UPD:
After re-installing pytorch3d via conda (which resulted in version 0.6.1) results for both cpu and cuda:0 became OK.

Not sure if further investigation is required, but thanks for your time anyway @gkioxari @nikhilaravi

UPD 2
Plane vertices coplanarity check fail is still happening on GPU from time to time

[gkioxari]

@lysukhin
The check for coplanar has an exposed eps value which you could increase. It's set by default to 1e-4.

pytorch3d/pytorch3d/ops/iou_box3d.py

Line 48 in 315f248

def _check_coplanar(boxes: torch.Tensor, eps: float = 1e-4) -> None: