DEV: reduce the size of the dev environment.yml

[jorisvandenbossche]

The main entry point for setting up a development environment for people to start contributing is to use the environment.yml file. Currently, this includes everything you would potentially need to run every single test, build every page of the docs, etc. This is of course nice (so people aren't surprised about something not working), but it also creates a huge and complex environment (more to download, taking up a lot of space, being slower to solve with conda).

I think that for 99% of the contributors you don't necessarily need all optional (test) dependencies. For example, we have some packages installed that are used for just a single test (eg geopandas, statsmodels, scikit-learn). pytorch was recently removed for this reason (#49796).
But the problem is that we don't only use the environment.yml for contributors setting up a dev env, but also for testing in our CI. So of course we if we make the environment.yml slimmer, then we need to solve this issue.

There is a PR to remove geopandas, but we should probably have a more general discussion about this, since it not just geopandas. From that PR, a comment by @mroeschke in #49994 (comment)

Yeah I would be partial to be consistent and remove geopandas everywhere if we also want to remove it from the environment.yml for consistency (CI dependency files, unit tests, any ASVs, any doc reference)

cc @MarcoGorelli @noatamir

[jorisvandenbossche]

consistency (CI dependency files, unit tests, any ASVs, any doc reference)

Looking at one set of test dependencies: the ones that are only used for test_downstream.py. I don't thiink that consistency is very important here, as those dependencies being present or not should not affect linting or asv. They are only used for a single test (and those will be skipped if the package is not installed). In addition, for the downstream tests, we already have a separate env yml file (actions-38-downstream_compat.yaml).
So I would argue that those dependencies could easily be removed from the main environment.yaml:

• statsmodels (also used for a few old whatsnew files, but those ipython code blocks can be converted to verbatim ones)
• scikit-learn
• seaborn (also used for a colormap in style.ipynb, but that could be replaces with a matplotlib one)
• pandas_datareader
• pandas_gbq (this is also used for our read_gbq/to_gbq, but those don't seem to have tests, I suppose they are only tested in the downstream package since they are slim wrappers)

Apart from those, the downstream tests also use:

• dask: used for downstream tests, and for a small section of the docs (in scale.rst) and a few array compat tests
• xarray: also used for a few tests for to_xarray

[MarcoGorelli]

I'd be all for removing those from the main environment file and just keeping them in the downstream compat file

If it allows 99% of contributors to run all the tests they need, I think that's good enough

[jorisvandenbossche]

Seems this is somewhat duplicate with #48828, although that issue goes further towards a really minimal yml file (which could be something in addition to the environment.yml file), while removing some deps might already be a good improvement for environment.yml usage as well.

(that said, I think having an actual environment_minimal.yaml would also be useful)

[MarcoGorelli]

Let's discuss in the next dev call, there's differing opinions, it'll be easier to resolve them

[jorisvandenbossche]

Let's discuss in the next dev call, there's differing opinions, it'll be easier to resolve them

Let's first try to just resolve it here, maybe those differing opinions turn out to be not that different if they are written down ;)

[mroeschke]

Agreed with reducing the size of environment.yml, but I would prefer to see it broken up into various dependency files that can be installed on top of each other #49798 (comment).

For consistency and given that the CI and the dependencies installed there are the ultimate gatekeeper for pull requests, it would be nice to make the sets of dependencies available in some form.

[jorisvandenbossche]

it would be nice to make the sets of dependencies available in some form.

All the deps needed to run test_downstream.py would still be available through the specific yml file used for those tests (i.e. currently actions-38-downstream_compat.yaml).
So you can still install them that way right now, and we could certainly also create an additional dep file specifically with the downstream deps to install on top of the main environment.yaml

[mroeschke]

Yeah if it's as simple as reorganizing our CI deps and environment.yml into a set of dependency files that both contributors and CI could use I would be very happy with that. It would allow contributors to install a smaller environment scoped to a contribution area and align with what the CI installs

[MarcoGorelli]

Just to check I've understood, would your aim be:

• you start with mamba env create --file environment.yml
• if you want to run downstream deps tests as well, you do mamba env update --file envs/test_downstream.yml
• if you want to run database ones, you do mamba env update --file envs/test_database.yml
  ?

[WillAyd]

If we wanted to leverage dockerhub we could create images like:

• pandas-dev:3.X-alpine
• pandas-dev:3.X-minimal
• pandas-dev:3.X-all
• pandas-dev:3.X-mamba-minimal
• pandas-dev:3.X-mamba-all

And some other variations. When I set up an alpine image for pandas with numpy, pytz, dateutil, hypothesis, cython and pytest it was around a 300 MB image. The 3.X-all is akin to our current Dockerfile and clocks in under 3 GB. mamba-all is what we have on gitpod which clock in a little over 6 GB

The advantage of docker would be users can pick what they want. I have a main development environment that has all of our dependencies, but setting that up for one-off tests in other versions again is a pain. Having an alpine image for another Python version would make that trivial to hop around

[mroeschke]

Just to check I've understood, would your aim be:

Yeah that was the flow I was imagining, such that users could install a subset of dependencies but also know that extra dependencies for development are specified in a certain place

The advantage of docker would be users can pick what they want.

Although it would be a larger development change, I would be supportive of moving in this direction. Especially since it allows users to use the same environment as what could be tested on the CI