fs::copy hangs on docker (Linux)

[h33p]

Running Fedora 32 with selinux set to non-enforcing, building a project with OpenCV.

I am building a rather weird docker setup, but what I ran into was hanging during build stage of opencv-rust, around here.

I expected to see this happen: build succeed rather quickly

Instead, this happened: execution froze on fs::copy call, and cpu got stuck running almost full speed.

My project was mounted using -v $PWD/project:/project:Z (name changed), opencv was manually cloned (for debugging) into the root of the docker image, at /opencv-rust.

Copies from /opencv-rust to /project don't work (example: /opencv-rust/bindings/cpp/opencv_4/aruco.cpp => /project/target/release/build/opencv-50ff47d79816a5ea/out/aruco.cpp)

File copies from /project to /opencv-rust work just fine (example: /project/target/release/build/opencv-50ff47d79816a5ea/out/xobjdetect_types.hpp => /opencv-rust/bindings/cpp/opencv_4/xobjdetect_types.hpp)

It appears to be an issue in the implementation of Linux's fs::copy, as implementing the more generic version above does not freeze the operation.

Meta

rustc --version --verbose:

rustc 1.45.2 (d3fb005a3 2020-07-31)
binary: rustc
commit-hash: d3fb005a39e62501b8b0b356166e515ae24e2e54
commit-date: 2020-07-31
host: x86_64-unknown-linux-gnu
release: 1.45.2
LLVM version: 10.0

[h33p]

Update: I reimplemented the Linux version, and the written count always gets stuck on 0 in the function.

[tavianator]

Any chance you can strace the process that gets stuck?

[h33p]

It's an endless copy_file_range(6, NULL, 11, NULL, 41205, 0) = 0

[h33p]

According to man, If the file offset of fd_in is at or past the end of file, no bytes are copied, and copy_file_range() returns zero., although the behaviour is odd. If I were to guess running this under docker does not return EXDEV, even though it is technically on external device (not in the view of the docker container though).

[h33p]

This particular case of 0 is not being handled in the current implementation, only the errors. Adding match arms and redirecting 0 to one of the errors to force using fallback mode could work. However, there may be cases where 0 gets returned legitimately?

[the8472]

The current implementation queries the size of the source file and then does a decrementing loop. Does the file in question get truncated by another thread while the copy is in progress?

[the8472]

Can you check which syscalls are used in the direction that works?

If I were to guess running this under docker does not return EXDEV, even though it is technically on external device (not in the view of the docker container though).

Recent kernels support cross device copy_file_range under certain circumstances, e.g. overlayfs can delegate to the underlying device and btrfs can copy between volumes. Bind mounts also should count as the same device. But maybe there's a bug in there in combination with selinux?

[cuviper]

If selinux is non-enforcing (per OP), that should rule it out.

[the8472]

It still seems really odd. The kernel's copy_file_range implementation itself does a lot of fallbacks internally when the preferred methods return a 0, i.e. 0 bytes copied ultimately lead to a splice operation from file to file which is not all that different than what we're doing in userspace. So 0 bytes being copied is quite unexpected for a non-empty file. There are lots of layers involved though and any of them could be the cause.

Adding yet another workaround is possible, but if possible I'd like to identify the root cause so we can report it to the responsible parties.

[tavianator]

Coreutils tests for a zero-byte copy_file_range() and falls back on a read()/write() loop if that happens: coreutils/coreutils@4b04a0c

Their code has a comment mentioning that this happens for /proc special files. I tried to reproduce it with

fs::copy("/proc/self/cmdline", "./foo");

but ran into another bug: stat() says that file is empty, so fs::copy() creates an empty file and doesn't copy anything. I think trusting st_size is fundamentally broken, for both this reason and the race that @the8472 mentioned.

[the8472]

@rustbot claim