Add temporary escape hatch for customization wheel generation?

[HDembinski]

I am trying to port my project pyhepmc scikit-hep/pyhepmc#57 to meson/meson-python.

I am building a Python extension module and a library (a dependency). I compile the library myself. What I would like to do is to compile the library as a shared library and ship it with the Python module. The library has to be a shared library, not a static library or compiled directly into my app, to respect the LGPL license (my project is released under a less restrictive license than the library). meson-python informs me that this is only supported for Linux at the moment.

        File "/Users/hdembinski/Extern/meson-python/.mesonpy/editable/install/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mesonpy/__init__.py", line 561, in build
          assert platform.system() == 'Linux', 'Bundling libraries in wheel is currently only supported in POSIX!'

I looked into the source code and the reason for this seems to be that there are no tools implemented yet to change RPATH on MacOS and Windows.

I am willing to live with a workaround until this is supported officially. Is it possible to implement an escape hatch as a temporary solution which allows me to put the shared library into the Python extension directory and allow me to run a custom command to change the RPATH in the Python extension module?

I would like to control the location where the library is installed, because I think that the RPATH in the Python extension module may actually support loading the library from the same directory where the extension module is located. What I have in mind is a undocumented configuration setting, that allows me to specify the location of the library and another setting to run a custom python script on the python extension module. Setting this should bypass the assert, too.

Or is there maybe another solution to my problem? I tried a few workarounds. I got pretty far with integrating the library via <dependency>.as_link_whole(), which works on Linux and MacOS, but fails on Windows. This still violates the LGPL, though, so it is not a proper solution.

Link to my meson.build https://github.com/scikit-hep/pyhepmc/blob/3d5372824e08e1cd4317eda6dd924ea649d68984/meson.build

[Brishen]

For Windows and Python>3.8 can you use os.add_dll_directory?

[rgommers]

For Windows and Python>3.8 can you use os.add_dll_directory?

We're doing something similar to this for SciPy, to ship libopenblas.dll inside the SciPy wheel. It's a little fiddly, but this is pretty robust by now: https://github.com/scipy/scipy/blob/main/tools/openblas_support.py#L199-L236. That _distributor_init.py is generated for wheel builds to be uploaded to PyPI only, and imported in scipy/__init__.py. Wheels are generated with cibuildwheel: https://github.com/scipy/scipy/blob/main/tools/wheels/cibw_before_build_win.sh

I looked into the source code and the reason for this seems to be that there are no tools implemented yet to change RPATH on MacOS

For macOS you can use delocate for this job, it works just like auditwheel.

tl;dr shipping shared libraries like this works on all platforms, but meson-python provides builtin support only on Linux, and you need a few more hacky tools on macOS/Windows.

[rgommers]

gh-260 adds support for macOS.

[HDembinski]

@rgommers Thank you for the comprehensive reply! I used scipy as a reference already for pyhepmc, the build scripts are sophisticated and well documented. I found anwers to most of my obstacles there. I missed the openblas support that you pointed out, thank you, I will look into it.

#260 adds support for macOS.

Excellent!

[FFY00]

@HDembinski with #260 being merged, is this still relevant for your use-case?

If so (or if anyone else in the future is reading this issue), achieving wheel customization can be done by creating a custom backend. Your custom backend would call meson-python, and then after getting the wheel, you are free to modify it, before passing it to the final user.

• pyproject.toml

[build-system]
build-backend = 'my_custom_backend'
backend-path = ['.']
requires = ['meson-python']

• my_custom_backend.py

from typing import Any, Dict, Optional

import mesonpy

def build_wheel(
    wheel_directory: str,
    config_settings: Optional[Dict[Any, Any]] = None,
    metadata_directory: Optional[str] = None,
) -> str:
    # do something before
    wheel = mesonpy.build_wheel(wheel_directory, config_settings, metadata_directory)
    # do something with the wheel
    return wheel

def __getattr__(name: str) -> Any:
    return getattr(mesonpy, name)

For more info about module-level __getattr__ see PEP 562

As such, I am gonna close this, but feel free to re-open if there's anything else you think I failed to consider, and please let me know if you have any further questions. Thanks!

---

Just noting this down for the future: this could be made a guide or tutorial for the docs.

[HDembinski]

This looks great and is exactly what I had in mind, thanks!

This whole customization becomes obsolete once support for embedding shared libs on Windows is also added. I haven't followed the development for a while, so I am not sure whether that has landed already.