shutil.which doesn't work in Docker container

[kentdlee]

Bug report

Bug description:

import shutil

target_fp = shutil.which(head_proc)
if target_fp is not None:
    perm = os.stat(target_fp)
    # This checks the permission bits as an extra check.
    # In a Docker container "which" was not returning
    # executable correctly when headproc was not in the
    # current working directory. This gets the user
    # permission value (rwx) and the remainder of 2
    # division (odd or even). If even, then executable
    # bit is not set and remainder is 0. bool(0) is False.
    # bool(1) is True.
    executable = bool(int(oct(perm.st_mode)[-3]) % 2)
    if not executable:
        target_fp = None
print(target_fp)

This only occurred in a Docker container, but was not a problem outside a container on an Ubuntu Linux box.

When head_proc is the name of a file in the current working directory (i.e. test.py) that is not executable, the shutil.which correctly returns None. However, when the current working directory is not where the file resides (i.e. subdir/test.py or /path/to/test.py) then shutil.which returns the filename and path incorrectly, indicating that it is executable, when it is not.

os.access also returns an incorrect value when mode is os.R_OK | os.X_OK when the current working directory is not where the file exists. If os.access is called on a file in the current working directory in a Docker container, it works as advertised. The code above demonstrates a work-around using os.stat that fixes it for now until this can be fixed.

CPython versions tested on:

3.11

Operating systems tested on:

Other

Linked PRs

• gh-132971: update shutil.which doc #133067

[zware]

Can you provide a minimal Dockerfile and/or test script to demonstrate this issue? I'm struggling to see how this could be a Python issue, but without a clear reproducer I could very easily be missing something.

[ericvsmith]

Also, it would be interesting to run stat [printed-value-of-target_fp] from a shell, and see what it returns.

[kentdlee]

Here is my Dockerfile.

FROM mcr.microsoft.com/vscode/devcontainers/base:ubuntu-22.04

ENV DEBIAN_FRONTEND=noninteractive

# Required packages for building Python via pyenv, see https://github.com/pyenv/pyenv/wiki#suggested-build-environment
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y \
    ca-certificates git \
    make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
    libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev \
    libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
    && rm -rf /var/lib/apt/lists/*

# Download pyenv to install Python
ENV PYENV_ROOT=/opt/pyenv
RUN git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT

# Compile dynamic bash extension to speed up pyenv
RUN cd $PYENV_ROOT && src/configure && make -C src

# Build Python 3.11.2 by default.
ARG python_version=3.11.2
# Keep Python source files in /usr/local/src
ENV PYTHON_BUILD_BUILD_PATH=/usr/local/src/
# Install a Python with a shared object
ENV PYTHON_CONFIGURE_OPTS="--enable-shared"
# Build Python and clean-up downloaded tarball
RUN /opt/pyenv/plugins/python-build/bin/python-build --verbose --keep ${python_version} /usr/local \
    && rm -fr /usr/local/src/Python-${python_version}.tar.*

# Install development tools required by Dragon
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y default-jdk doxygen git openssl sudo unzip util-linux clang-format \
    && rm -rf /var/lib/apt/lists/*

# Install SRMStoFigs dependencies
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y imagemagick transfig \
    && rm -rf /var/lib/apt/lists/*
# See https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion
RUN sed -i '/disable ghostscript format types/,+6d' /etc/ImageMagick-6/policy.xml

RUN curl https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.bash -o ~/.git-completion.bash
COPY git_completion.bash /tmp/git_completion.bash
RUN cat /tmp/git_completion.bash >> ~/.bashrc

# Install SRMStoFigs (required to build Dragon docs)
RUN cd /usr/local/src && git clone https://github.com/kentdlee/SRMStoFigs.git
RUN cd /usr/local/src/SRMStoFigs \
    && make -B srmstofigs CC=gcc \
    && install -D -m 755 srmstofigs srms2pdf srms2png /usr/local/bin

# Additional packages for convenience
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y \
    debianutils gdb git-lfs git-man graphviz man \
    manpages manpages-dev manpages-posix manpages-posix-dev ssh psmisc \
    rsync sshpass tree vim xfsprogs \
    && rm -rf /var/lib/apt/lists/*

# Python packages required by Dragon
COPY requirements.txt /usr/src/requirements.txt

# Zarr/numcodecs requires special attention to work around a bug in numcodecs (w/o going to the latest Zarr)
RUN /usr/local/bin/python3 -m ensurepip \
    && /usr/local/bin/python3 -m pip install --no-cache-dir --upgrade pip \
    && /usr/local/bin/python3 -m pip install --no-cache-dir -r /usr/src/requirements.txt

# Packages for building dragon multinode dependencies
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y \
    automake autopoint flex bison cmake libtool-bin elfutils \
    libboost-all-dev libelf-dev libssh2-1-dev libarchive-dev libdwarf-dev \
    libdw-dev libiberty-dev libacl1-dev \
    && rm -rf /var/lib/apt/lists/*


# Run script to allow non-root vscode user to map to your own uid
# by replacing uid with your own (result of `id -u <username>`)
# on host system
#  COPY library-scripts/common-debian.sh /tmp/library-scripts/
#  RUN apt-get update \
#       && bash /tmp/library-scripts/common-debian.sh false vscode \
#                uid automatic false false false

# We must expose a port to run the Jupyter Notebook example.
EXPOSE 8888

[kentdlee]

devcontainer.json contents

{
    "build": {
        "dockerfile": "Dockerfile"
    },
    "runArgs": [
        "--privileged", // so gdb can attach to processes
        "--ulimit", // These two lines allow core files in the container.
        "core=-1", // Helpful in debugging.
        "--shm-size=6gb", // 4GB user pool, 1GB inf pool, 1GB extra pool
        "--cpus=2", // explicit hard limits for CPUs and mem
        "--memory=8GB",
        "--cap-add=SYS_PTRACE"
    ],
    "onCreateCommand": [],
    // remove the slow git status from the prompt.
    "postStartCommand": "git config codespaces-theme.hide-status 1",
    "settings": {
        "terminal.integrated.profiles.linux": {
            "bash (login)": {
                "path": "/bin/bash",
                "args": [
                    "-li"
                ]
            }
        },
        "terminal.integrated.defaultProfile.linux": "bash (login)"
    },
    "extensions": [
        "donjayamanne.git-extension-pack",
        "mhutchie.git-graph",
        "ms-python.python",
        "ms-vscode.cpptools",
        "ms-vscode.cpptools-extension-pack",
        "ms-vscode.cpptools-themes",
        "njpwerner.autodocstring", // allow automatic docstrings in Python
        "jebbs.plantuml" // support .puml UML diagrams for the docs
    ]
    // Uncomment if you set up vscode in Dockerfile and want
    // to run as non-root vscode in devcontainer
    // "remoteUser": "vscode"
}

[kentdlee]

I am working on an open source project, but we need to update the open source so I just copied the contents of Dockerfile and devcontainer.json for now. I hope this works for you. As for a reproducer, here you go. Just touch a file called test/test.py. There doesn't need to be anything in the file. Then place this reproducer in the directory that contains the test directory.

import os
import shutil

def check_file(head_proc):
    target_fp = shutil.which(head_proc)
    print(target_fp)
    result = os.access(head_proc, mode=os.X_OK)
    print(result)
    perm = os.stat(head_proc)
    print(perm)
    print(oct(perm.st_mode))
    try:
        print(f"{os.stat(target_fp)=}")
    except Exception as ex:
        print(f"Caught exception: {ex}")

def main():
    print("While cwd not in the directory test.")
    print("************************************")
    check_file("test/test.py")
    os.chdir("test")
    print()
    print("While cwd in the directory test.")
    print("************************************")
    check_file("test.py")
    print()
    print("While cwd in the directory test.")
    print("************************************")
    check_file("../test/test.py")

if __name__ == "__main__":
    main()

Here is the output I see when I run it in a Docker container. On a regular Linux box I get consistent results that test.py is not executable no matter what my current working directory is, but in a Docker container, this is what I see. I will amend what I said about current working directory though. shutil.which does not return the correct output if any path is provided and not just the filename by itself from the current working directory. See the third call to check_file for passing a path while in the same directory as test.py. So the issue seems to be something with passing a path at all to which. On a stand-alone Linux install (SUSE) all three calls to which print None as the result.

(_env) vscode ➜ /workspaces/hpc-pe-dragon-dragon $ python3 reproducer.py 
While cwd not in the directory test.
************************************
test/test.py
True
os.stat_result(st_mode=33188, st_ino=92919373, st_dev=42, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1745636134, st_mtime=1745636113, st_ctime=1745636133)
0o100644
os.stat(target_fp)=os.stat_result(st_mode=33188, st_ino=92919373, st_dev=42, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1745636134, st_mtime=1745636113, st_ctime=1745636133)

While cwd in the directory test.
************************************
None
True
os.stat_result(st_mode=33188, st_ino=92919373, st_dev=42, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1745636134, st_mtime=1745636113, st_ctime=1745636133)
0o100644
Caught exception: stat: path should be string, bytes, os.PathLike or integer, not NoneType

While cwd in the directory test.
************************************
../test/test.py
True
os.stat_result(st_mode=33188, st_ino=92919373, st_dev=42, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1745636134, st_mtime=1745636113, st_ctime=1745636133)
0o100644
os.stat(target_fp)=os.stat_result(st_mode=33188, st_ino=92919373, st_dev=42, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1745636134, st_mtime=1745636113, st_ctime=1745636133)

It must be the case that I was misunderstanding os.access. It prints True for every call here and also prints True on the Linux box. os.access must be looking at the path to the file and not the file itself. The documentation says path is checked, but I had assumed path would include the filename if provided since a permission bit was provided in mode. So I'll retract what I said about os.access not working correctly. But clearly something is going on with shutil.which.

Please let me know if you need anything further. Thanks for looking into this.

[kentdlee]

Results of running stat both in test directory and above test directory.

(_env) vscode ➜ /workspaces/hpc-pe-dragon-dragon/test $ stat test.py
  File: test.py
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 2ah/42d Inode: 92919373    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  vscode)   Gid: ( 1000/  vscode)
Access: 2025-04-26 02:55:13.277581423 +0000
Modify: 2025-04-26 02:55:13.277582048 +0000
Change: 2025-04-26 02:55:33.637132759 +0000
 Birth: -
(_env) vscode ➜ /workspaces/hpc-pe-dragon-dragon/test $ cd ..
(_env) vscode ➜ /workspaces/hpc-pe-dragon-dragon $ stat test/test.py
  File: test/test.py
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 2ah/42d Inode: 92919373    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  vscode)   Gid: ( 1000/  vscode)
Access: 2025-04-26 02:55:13.277581423 +0000
Modify: 2025-04-26 02:55:13.277582048 +0000
Change: 2025-04-26 02:55:33.637132759 +0000
 Birth: -
(_env) vscode ➜ /workspaces/hpc-pe-dragon-dragon $ 

[ericvsmith]

What is the result of os.stat(target_fp) for both cases?

[zware]

It looks like the documentation for which needs an update; the implementation has a comment specifying that path is ignored if the given cmd contains a directory part, see here:

cpython/Lib/shutil.py

Lines 1515 to 1520 in 8fdc106

	# If we're given a path with a directory part, look it up directly rather
	# than referring to PATH directories. This includes checking relative to
	# the current directory, e.g. ./script
	dirname, cmd = os.path.split(cmd)
	if dirname:
	path = [dirname]

This is not mentioned in the documentation, but should be.

If you want to use which to check a filename that is not found in a directory on PATH, you need to include a directory (even if that directory is ./). Note that this matches the behavior of the standalone which command.