BUILD: 2.1.0 not installable via installer because of duplicate files in wheel

[mgorny]

Installation check

• I have read the installation guide.

Platform

Linux-6.4.7-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.38

Installation Method

Built from source

pandas Version

2.1.0

Python Version

3.11.5

Installation Logs

Build log

$ python3.11 -m build -w
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.33,<3, meson-python==0.13.1, meson==1.0.1, numpy>=1.22.4; python_version>='3.12', oldest-supported-numpy>=2022.8.16; python_version<'3.12', versioneer[toml], wheel)
* Getting build dependencies for wheel...
* Building wheel...
+ meson setup /tmp/pandas-2.1.0 /tmp/pandas-2.1.0/.mesonpy-vs0fc5ea/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/tmp/pandas-2.1.0/.mesonpy-vs0fc5ea/build/meson-python-native-file.ini
The Meson build system
Version: 1.0.1
Source dir: /tmp/pandas-2.1.0
Build dir: /tmp/pandas-2.1.0/.mesonpy-vs0fc5ea/build
Build type: native build
Project name: pandas
Project version: 2.1.0
C compiler for the host machine: ccache cc (gcc 13.2.1 "cc (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826")
C linker for the host machine: cc ld.bfd 2.41
C++ compiler for the host machine: ccache c++ (gcc 13.2.1 "c++ (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826")
C++ linker for the host machine: c++ ld.bfd 2.41
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/tmp/build-env-qqfvpycz/bin/python)
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Build targets in project: 53

pandas 2.1.0

  User defined options
    Native files: /tmp/pandas-2.1.0/.mesonpy-vs0fc5ea/build/meson-python-native-file.ini
    buildtype   : release
    b_ndebug    : if-release
    b_vscrt     : md

Found ninja-1.11.1 at /usr/bin/ninja
+ /usr/bin/ninja
[151/151] Linking target pandas/_libs/join.cpython-311-x86_64-linux-gnu.so
+ meson install --quiet --no-rebuild --destdir /tmp/pandas-2.1.0/.mesonpy-vs0fc5ea/install
[54/65] pandas/_libsmeson-python: warning: Duplicate name: 'pandas/_libs/__init__.py'
meson-python: warning: Duplicate name: 'pandas/_libs/tslibs/__init__.py'
[65/65] pandas/util
Successfully built pandas-2.1.0-cp311-cp311-linux_x86_64.whl
$ python3.11 -m installer --destdir /tmp/z dist/pandas-2.1.0-cp311-cp311-linux_x86_64.whl 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.11/site-packages/installer/__main__.py", line 98, in <module>
    _main(sys.argv[1:], "python -m installer")
  File "/usr/lib/python3.11/site-packages/installer/__main__.py", line 94, in _main
    installer.install(source, destination, {})
  File "/usr/lib/python3.11/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: /tmp/z/usr/lib/python3.11/site-packages/pandas/_libs/__init__.py
$ unzip -l dist/pandas-2.1.0-cp311-cp311-linux_x86_64.whl  | grep _libs/__init
      691  08-31-2023 04:57   pandas/_libs/__init__.py
      691  08-31-2023 04:57   pandas/_libs/__init__.py

[thesamesam]

Note that pip will likely work as it seems to ignore duplicates, while installer rejects them.

[lithomas1]

Do you know if this issue is present on the uploaded wheels to PyPI?

Inspecting this wheel (the manylinux wheel for Python 3.11) https://files.pythonhosted.org/packages/d9/26/895a49ebddb4211f2d777150f38ef9e538deff6df7e179a3624c663efc98/pandas-2.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl,

I only see 1 __init__.py in _libs.

[mgorny]

Indeed the wheel from PyPI looks and works fine. I'll try digging in.

[mgorny]

Apparently auditwheel repair run as part of cibuildwheel pipeline repacks the wheel, effectively hiding the problem.

[mgorny]

Guessing by the warnings from meson install, I've removed the explicit install calls for __init__.py, i.e.:

diff --git a/pandas/_libs/meson.build b/pandas/_libs/meson.build
index f302c649bc..ddce9ea2d6 100644
--- a/pandas/_libs/meson.build
+++ b/pandas/_libs/meson.build
@@ -113,8 +113,4 @@ foreach ext_name, ext_dict : libs_sources
     )
 endforeach
 
-py.install_sources('__init__.py',
-                    pure: false,
-                    subdir: 'pandas/_libs')
-
 subdir('window')
diff --git a/pandas/_libs/tslibs/meson.build b/pandas/_libs/tslibs/meson.build
index 14d2eef46d..a862345c3a 100644
--- a/pandas/_libs/tslibs/meson.build
+++ b/pandas/_libs/tslibs/meson.build
@@ -30,7 +30,3 @@ foreach ext_name, ext_dict : tslibs_sources
         install: true
     )
 endforeach
-
-py.install_sources('__init__.py',
-                    pure: false,
-                    subdir: 'pandas/_libs/tslibs')

and got a correct wheel as a result.

[eli-schwartz]

pandas/pandas/meson.build

Lines 27 to 44 in 6884d1c

	subdirs_list = [
	'_config',
	'_libs',
	'_testing',
	'api',
	'arrays',
	'compat',
	'core',
	'errors',
	'io',
	'plotting',
	'tests',
	'tseries',
	'util'
	]
	foreach subdir: subdirs_list
	install_subdir(subdir, install_dir: py.get_install_dir(pure: false) / 'pandas')
	endforeach

They're installed in bulk, and then manually installed again.

[lithomas1]

No this if for _libs and _libs/tslibs, which aren't part of the subdirs_list I think.

[eli-schwartz]

I see _libs in that list, though.

[lithomas1]

My bad, I must have typo'ed there.

Thanks for the catch!