Supporting arm64, aarch64, and other platforms.

[TomAugspurger]

There are some desires / efforts to support other platforms like arm64 / aarch64 (#30641). I might have missed it, but I don't know that we've had a real discussion on if / how we are going to support those.

I'm not against adding support for either of these, but we should do it deliberately. I think that us uploading wheels / conda packages implies some kind of official blessing that we support those platforms, and will fix bugs specific to those platforms when they arise. These platform specific issues are extremely hard for community-maintained open-source projects to manage (we've had enough issues with 32-bit Windows / Linux). Contributors and maintainers don't have access to hardware and are stuck trying to debug things through CI.

Let's use this issue to discuss

1. Do we want to support these two platforms? (I think yes to both)
2. What are the requirements for us to add a platform?

• Freely available CI infrastructure
• NumPy supports it (any other deps? Cython? Arrow at some point?)
• Some level of popularity? We drop Python versions based on popularity, but this seems hard to measure for platforms we don't already officially support.

3. We need a person / team who will champion platform-specific issues. We can ping that person or team when CI / packaging issues arise with that platform.

I think we can discuss this more on our monthly dev call next week.

cc @pandas-dev/pandas-core & @odidev and @ossdev07 in particular.

[jreback]

+1 on these arches, even though this would add to our CI + release burden. Do we have stats anywhere that shows 'popularity' anywhere? does numpy support these?

[TomAugspurger]

NumPy does aarrch64 stuff at https://github.com/numpy/numpy/blob/master/shippable.yml, but doesn't currently have wheels (numpy/numpy#14886).

Not sure about arm64.

[cphang99]

The libreML project builds SDKs for machine learning applications using the BuildStream integration tool.

We have recently gained capacity for arm64 builds in our CI infrastructure, e.g. https://gitlab.com/libreml/libreml/-/merge_requests/253 with the potential of adding other architectures if hardware permits. We already have the capacity to track new releases of pandas, and to test this in our CI, in an automated fashion.

We would be interested in supporting this effort whereever we can.

[TomAugspurger]

Thanks @cphang99, will take a look. FWIW, we're fairly heavily invested in multibuild / https://github.com/MacPython/pandas-wheels for our wheel building.

---

@mattip can you share the status on what's needed for NumPy to upload arm64 wheels to PyPI? Is it completely blocked by pypa/manylinux#542?

I also see that @odidev added support for arm64 to SciPy in scipy/scipy#11867 (cc @rgommers). I feel like there's opportunity for coordinating efforts around building wheels here (and perhaps you already are).

[mattip]

There is MacPython/numpy-wheels#74, which is blocked right now on a failing aarch64 test. That would be solved by being able to move past manylinux2014 (of which pypy/manylinux#542 is a part) or by simply skipping the test, numpy/numpy#16214.

We are trying to coordinate. In discussions, it is felt there is no need to require a numpy wheel before releasing a scipy wheel, since it is much easier to locally build numpy than to build scipy.

[rgommers]

NumPy is now unblocked, we should have wheels for 1.19.0 (branching soon) it looks like (thanks @mattip!).

I feel like there's opportunity for coordinating efforts around building wheels here (and perhaps you already are).

As far as I can tell, it's similar to other wheel production efforts - we rely on the same multibuild machinery, and adopt the same strategies for releasing wheels (without much formal coordination, just copy what works). Pushing manylinux2014 wheels to PyPI whenever you have them ready sounds great to me.

[odidev]

Numpy wheel for AArch64 is available now. Please have a look at this

[mattkanwisher]

A lot of machine vision is happening on the edge with Linux boxes. We are seeing hour+ installs of packages like pandas. Would be amazing to have aarch64 wheels

[mattip]

The work is taking place in this PR MacPython/pandas-wheels#99

[amerry]

Pandas 1.1.5 provided manylinux_aarch64 wheels for Pythons 3.6 through 3.8, which was great, thank you for that! The 1.2 release, however, doesn't include a manylinux_aarch64 wheel for Python 3.7, though - only for 3.8. Was this deliberate?

There is also no manylinux_aarch64 wheel for Python 3.9 (neither in 1.1.5 nor 1.2) - not sure whether this is deliberate either.