Python 3.12, pyproject.toml, distutils and a new build system

[oscarbenjamin]

Currently python-flint builds using a setup.py and also uses distutils and numpy.distutils (see #4). As of Python 3.12 the stdlib distutils module is removed and also numpy is removing its own distutils module. This means that python-flint needs to move to a new build system somehow. Also setup.py should be removed and replaced with pyproject.toml and it probably makes sense to do that at the same time.

There are a few options for build system:

1. Try to use setuptools instead of distutils.
2. Switch to meson and meson-python.
3. Use cmake

Various Python projects seem to be choosing one of these three options. For pure Python projects there are other options like poetry, hatch, etc. Some of those can handle extension module building but I am not sure how well any of them works. The people who seem to know most about packaging extension modules seem to suggest either meson or cmake. In particular meson is being used by SciPy now and will be for NumPy soon (if not already).

To me switching to meson seems like the best long term option. I made a start on that here:
mesonbuild/meson-python#410
I discussed lots of things there like editable installs etc that make things a bit more complicated but the most important thing is just to have a build system that can produce wheels either for local install or upload to PyPI. Restricting to just producing wheels means that it would probably not be too hard to get something working with meson but the work I started here just needs to be carried through:
https://github.com/oscarbenjamin/mesontest

I don't know much about how using cmake would work here or what the potential upsides/downsides are.

Potentially the quickest fix for the removal of distutils is just to replace distutils with setuptools. It is probably not hard for someone to test that:

• Replace distutils with setuptools in setup.py
• Fix any immediate problems until you have a local build
• Push up to a PR in this repo and see if CI can build for all platforms and architectures.

One thing that might be a problem with setuptools is the use of MinGW compilers on Windows but this is tested in CI so we should know if it is working or not. My understanding is that numpy.distutils has special support for MinGW that is not in distutils or setuptools. If that means that the Windows build fails with setuptools then it probably means that python-flint just can't use setuptools.

Switching somehow to MSVC rather than MinGW would be good but I have already explored the options for doing that with GMP and I think we just have to give up on that. What might be possible is using MinGW to make a fat build of GMP and then using MSVC to compile everything else or perhaps just for the python-tools extension module. I don't know enough about MSVC to investigate that option so someone else would need to explore that. If we could get MSVC to pick up the MinGW compiled GMP then that might make it more possible to use setuptools or probably makes things easier in some sense for any of the build system choices.

Along with the new build system we should replace setup.py with pyproject.toml which is how the build system should be configured for the project in modern Python packaging. Other things can also be moved into pyproject.toml:

• Build-time dependencies (Missing dependencies #24) - I don't see how to add these in setup.py.
• All of the cibuildwheel configuration variables that are currently in the CI config and some of the scripts in bin/.

It possibly makes sense to create a pyproject.toml before attempting to use a new build system because there needs to be a pyproject.toml in order for any standardised build system to be used and for cibuildwheel to work without setup.py.

[eli-schwartz]

For pure Python projects there are other options like poetry, hatch, etc. Some of those can handle extension module building but I am not sure how well any of them works.

Poetry-core has undocumented support for defining a .py file that contains an arbitrary build hook which you then use to bring your own build system (typically setuptools). I don't think it adds anything here.

Hatchling has the concept of build plugins which is sort of the same thing except documented -- you write your own build system, or wrap an existing one, and then you write code to tell hatchling that the resulting files were created.

Realistically in both cases you are going to end up using something like meson, autotools or cmake anyway. I'm skeptical that either one provides value over just using the build system directly.

[oscarbenjamin]

I'm skeptical that either one provides value over just using the build system directly.

Agreed. The various bells and whistles that different Python packaging systems provide are completely dwarfed in python-flint's case by the much more important problem of just being able to build the extension module for the various platforms. Some of these systems have some support for building relatively simple Python packages with extension modules and python-flint is not really that complex having only a single Cython module. The critical issues are though:

• Need to build the C dependencies.
• Need to use MinGW on Windows at least for GMP if not necessarily for all dependencies and the flint._flint extension module.

We have a script bin/build_depependencies_unix.sh that can build the dependencies including on Windows with MinGW. Provided the python package build system can tell the compiler to pick up these libs then any build system is fine and we do not necessarily need it to be able to build the libs. It would be nice if it could build the libs and that is what I tried to do in:
https://github.com/oscarbenjamin/mesontest
That setup would make it so that if pip install python-flint (or pip install . etc) needs to build from source then meson could build GMP etc which is nice but not critical: we can just tell users to build the dependencies first or arrange to build them first when building the wheels (which is already how it works now with setup.py).

The MinGW issue is critical though. We cannot use MSVC for GMP and so we either use MinGW for everything or somehow have a mixed build system. Any extension module build system needs to be able to use MinGW unless someone can figure out how to tell MSVC to pick up the MinGW libs.

Note also that currently on Windows python-flint and all DLLs in the wheel link against msvcrt.dll rather than ucrt. I have an old PR #41 that tried to use the MinGW ucrt toolchain but somehow it didn't work. If using MSVC then either it needs to use msvcrt.dll as well or we need to figure out the ucrt build issue.

So unless someone wants to do a bunch of work for Windows then the choice of build system is really just: can it build extension modules with MinGW?

Note that all the same issues around GMP, MinGW etc apply to gmpy2 and to SciPy (not GMP but needs MinGW for fortran libs). I opened an issue with gmpy2 to discuss how to do this:
aleaxit/gmpy#352

[casevh]

What might be possible is using MinGW to make a fat build of GMP and then using MSVC to compile everything else or perhaps just for the python-tools extension module. I don't know enough about MSVC to investigate that option so someone else would need to explore that. If we could get MSVC to pick up the MinGW compiled GMP then that might make it more possible to use setuptools or probably makes things easier in some sense for any of the build system choices.

gmpy2 supports Cython by providing a C-API. I have a report that someone has successfully used MSVC to compile a Cython extension and link to the MinGW versions of GMP, MPFR, and MPC included with gmpy2. See aleaxit/gmpy#320 (comment)

Note also that currently on Windows python-flint and all DLLs in the wheel link against msvcrt.dll rather than ucrt. I have an old PR #41 that tried to use the MinGW ucrt toolchain but somehow it didn't work. If using MSVC then either it needs to use msvcrt.dll as well or we need to figure out the ucrt build issue.

I have successfully built gmpy2 with (a version of) the MinGW ucrt toolchain. It appeared to work but hasn't been extensively tested.

That setup would make it so that if pip install python-flint (or pip install . etc) needs to build from source then meson could build GMP etc which is nice but not critical: we can just tell users to build the dependencies first or arrange to build them first when building the wheels (which is already how it works now with setup.py).

I include the compiled Windows dependencies in gmpy2's GitHub repository. They are not included in the standard source distribution. To compile from source, you'd either need to install a previous version of gmpy2 or git clone ....

I don't know the impact of sharing libraries between python-flint and gmpy2 and sage-math and ??? I suspect there will be some subtleties. Especially with library-wide settings (MPFR and precision, user-definable memory allocation memory hooks for GMP that are automatically used by MPFR and MPC, should mpbitcnt be long or long long on 64-bit Windows). And then add a future with no GIL and multiple interpreters.

Slight different topic, GMP 6.3 with support for Mac OSX and some other enhancements has just been released. Are you planning to use that version? I think an update to MPFR is also planned.

[rgommers]

@oscarbenjamin I think your mesontest is pretty close to the finish line already, right? With the DLL loading on Windows being the main issue to the immediate problem at hand of being able to publish working wheels for Python 3.12?

There are more topics discussed in mesonbuild/meson-python#410 that will require time to unravel and make polished, but being able to vendor a DLL and load it on import should be doable.

[oscarbenjamin]

I don't know the impact of sharing libraries between python-flint and gmpy2 and sage-math and ??? I suspect there will be some subtleties. Especially with library-wide settings (MPFR and precision, user-definable memory allocation memory hooks for GMP that are automatically used by MPFR and MPC, should mpbitcnt be long or long long on 64-bit Windows). And then add a future with no GIL and multiple interpreters.

I'm really not sure how viable this is and I don't know enough of the internals of GMP and MPFR here to know what the problems might be. Specifically in python-flint's case MPC is not used and MPFR is not used directly but somehow Flint uses it and I am not sure what parts are used or what they are used for.

It would definitely be nice to be able to share work here and avoid duplicating shared libraries at runtime if e.g. gmpy2 and python-flint are used simultaneously. Being constrained to support the same versions of GMP and MPFR as gmpy2 would not be a problem for python-flint as long as gmpy2 is managing any patches or whatever are needed for them. If gmpy2 could handle the GMP build so that python-flint could use MSVC then that would be very nice.

I think that Flint itself requires that GMP be built with 32 bit limbs on a 32 bit system and 64 bit limbs on a 64 bit system regardless of OS or sizeof(long) etc. So if Flint is there than I think that some ways of building GMP are ruled out.

Slight different topic, GMP 6.3 with support for Mac OSX and some other enhancements has just been released. Are you planning to use that version?

It is already used in the 0.4.x releases:

python-flint/bin/build_variables.sh

Lines 16 to 21 in 71bf110

	GMPVER=6.3.0
	YASMVER=1.3.0
	MPIRVER=3.0.0
	MPFRVER=4.1.0
	FLINTVER=2.9.0
	ARBVER=2.23.0

YASM and MPIR are not used but the other versions listed there are what is built and bundled in the wheels.

[oscarbenjamin]

I think your mesontest is pretty close to the finish line already, right?

Yes, I think so. Someone just needs to take the time to add all the pieces, test and fix on every OS, make it all work in CI, update dev docs etc. If no major problem emerges during that then hopefully it is just a question of someone finding the time to work it through (I don't have time right now).

With the DLL loading on Windows being the main issue to the immediate problem at hand of being able to publish working wheels for Python 3.12?

The immediate problem is definitely solvable if only in a hacky way. Ultimately we are talking about building one single extension module. All distutils is really doing for us putting together a single command line call to gcc. Unfortunately cibuildwheel hides that command line but it is just something like:

gcc -I python_include_dir src/pyflint.c -o build/flint/_flint.pyd

If we really have to then it is not hard to manually reproduce everything that distutils does in python-flint's case.

[oscarbenjamin]

Now that Python 3.12 is released this becomes more urgent.

[casevh]

Latest update. I have compiled GMP, MPFR, and MPC using mingw64 and linking them to mscvrt. I then followed the documentation at https://github.com/GBillotey/Fractalshades to create MSVC compatible .lib files. I compiled gmpy2 with MSVC and linked to the DLLs compiled by mingw64. The gmpy2 tests passed.

There are more tests that I want to run. I plan to bundle all the required files into the Window binary wheels so it should be as simple as install gmpy2 and using MSVC.

I modify the GMP to make mp_bitcnt_t == long long' instead of long. This provide 64-bit addressing for bit indexing but does impact external code that assumes mp_bitcnt_tislongon Windows. The GMP docs do say thatmp_bitcnt_tcould becomelong long` in the future. Does this matter to flint?

Cython files should also be compatible with the GMP, MPFR, and MPC libraries.

Could this be a viable solution for flint?

[oscarbenjamin]

I compiled gmpy2 with MSVC and linked to the DLLs compiled by mingw64. The gmpy2 tests passed.

Nice!

Do you have this running in CI at all or is it just done locally?

I modify the GMP to make mp_bitcnt_t == long long' instead of long

Currently python-flint has:

python-flint/src/flint/flintlib/flint.pxd

Line 31 in dbbf8bb

ctypedef unsigned long mp_bitcnt_t

So I guess some changes would be needed in some places to build against that.

Could this be a viable solution for flint?

Maybe. I want everything running in CI so we will need to figure out how to use MSVC in CI. Probably that's just a case of removing the hacky workarounds that patch in mingw64 though:

python-flint/bin/cibw_before_all_windows.sh

Lines 8 to 14 in dbbf8bb

	#
	# Make a setup.cfg to specify compiling with mingw64 (even though it says
	# mingw32...)
	#
	echo '[build]' > setup.cfg
	echo 'compiler = mingw32' >> setup.cfg
	cat setup.cfg