Windows: Provide an option to disable search in CWD on `shutil.which`

[naveen521kk]

I would like to search for an executable in the PATH but not from the current working directory (cwd). By default, as documented in shutil.which, it prepends the cwd before PATH, so it's not possible to get the expected executable.

For example, I have a whoami.exe in the cwd but I would like to get the one from the system32 directory; there's no way to do so using shutil.which.

>>> import shutil
>>> shutil.which('whoami')
'.\\whoami.EXE'

I think it makes sense for that to be the default behaviour since that's how it is on Windows. I would like to have a parameter to disable this behaviour in shutil.which.

Thanks!

[eryksun]

WinAPI CreateProcessW() and the CMD shell respect NeedCurrentDirectoryForExePathW(). This function is false only if the ExeName argument contains no backslashes and the environment variable NoDefaultCurrentDirectoryInExePath is defined. The API should be used instead of the environment variable, in case the function is modified to depend on a different variable, registry setting, or a compatibility mode.

Note that NeedCurrentDirectoryForExePathW() does not handle forward slash as a path separator, as is actually the case with many functions that work with file paths in Windows. The ExeName argument has to be normalized to replace forward slashes with backslashes if implementing a Windows-style search matters. In the Windows API, if a filename has no root or drive, the system's search routine tries to resolve it against each directory in the search path, until the first match, even if the filename contains one or more backslashes. In this case, the current directory is the second directory that's checked, after the application directory. In POSIX, as far as I know, a relative path that contains one or more slashes is always resolved against the current directory.

[naveen521kk]

Are you suggesting that NeedCurrentDirectoryForExePathW should be used rather than providing a parameter to override this behaviour?

[eryksun]

Yes, a long time ago I suggested that shutil.which() should check NeedCurrentDirectoryForExePathW() (added in Windows Vista, circa 2006) to determine whether or not the current working directory should be included. This would match the behavior of both CreateProcessW() and the CMD shell, as used by subprocess.Popen with both shell=False (default) and shell=True. However, nothing came of the suggestion.

Since shutil.which() implements a POSIX style search, the check only applies when the cmd argument contains no slashes or backslashes, as determined by os.path.dirname().

[naveen521kk]

Thanks for the reply! Using that API makes sense as it matches with how cmd/CreateProcess works, though I think it would be better to have a flag in shutil.which which disables this behaviour. What do you think?

[lazka]

One difference between current shutil.which() and CreateProcessW(), is that the later always prefers binaries in the system directory, so things like tar, curl, bash would then never return the version in PATH, but the Windows shipped version. I depend on this in various scripts, though not sure how common that is..

[eryksun]

CreateProcessW() ... always prefers binaries in the system directory

CreateProcessW() first checks the application directory, even if the path is a relative path that has one or more backslashes (e.g. r'bin\bash.exe'). Next, if NeedCurrentDirectoryForExePathW() is true, it checks the current directory. Next it checks the "System32" directory and the "Windows" directory. If the command still isn't found, it checks the PATH directories, which can explicitly include the current directory as ".". When checking a directory, it looks for the given name and also the name with ".EXE" appended. It doesn't use PATHEXT.

shutil.which() is closer to the CMD shell's executable file search, since it uses PATHEXT and doesn't prefer the application directory and system directories over PATH. Starting with Windows Vista, CMD also calls NeedCurrentDirectoryForExePathW() to determine whether to implicitly include the current directory in the search path.

That said, shutil.which() uses PATHEXT differently from the shell, and thus differently from os.system() and subprocess.Popen with shell=True. CMD uses PATHEXT to expand the search with a set of extensions to try appending in addition to checking for the given name. It doesn't use it as a security policy that limits the search to just an allowed set of file types.