ENH: add additional extras_require sections for optional dependencies

[person142]

Is your feature request related to a problem?

It would be nice to have more extras_require sections in the setup.py for optional dependencies.

Describe the solution you'd like

I'd like to see more sections added here:

https://github.com/pandas-dev/pandas/blob/master/setup.py#L743

for optional dependencies; e.g. pandas[s3] for Pandas + everything you need to write to S3 (just for the sake of example; this is probably not the right division). I found a similar request here: #35206. (Sorry if a previous issue exists and I missed it.)

API breaking implications

None

Describe alternatives you've considered

Pin all desired optional dependencies.

Additional context

The idea is that if you're using a tool like pip-compile, then currently you have to pin all of these requirements, even if they are only used transitively by Pandas. It would be nice to do say pandas[s3] in your requirements.in and have s3fs get added to the compiled requirements.

Happy to make a PR if folks don't object to the idea.

[JMBurley]

Idea is still relevant, but since initial request pandas has refactored to more modern dependency mgmt and the necessary extras_require is held in setup.cfg: https://github.com/pandas-dev/pandas/blob/master/setup.cfg#L48.

I'll fix to insert current recommended dependencies

[JMBurley]

@mroeschke PR #47336 resolves some of the issues discussed here.

It would also be trivial to repeat my code to provide functional-level extras like pandas s3, gcp (etc...) if package ranges are agreed upon. I can do that over the next few weeks.

(Well, pandas s3 uses s3fs which means a hideous set of dependency problems derived from their use in turn of aiobotocore (in s3fs>=0.5.0) which is typically incompatible with any other package using boto3 but that's not for pandas to resolve. I'd want to check with pandas leadership if s3fs>=0.5 is needed for any reason and strongly suggest pinning s3fs ~=0.4 until that dependency mess is resolved. Summary here)

[mroeschke]

Thanks for initiating this! I think there may need to be a broader discussion with the core team before we make any PR on organizing the optional dependencies. I think the broad steps will be.

1. Decide on the correct logical grouping of optional dependencies
2. Establish the groupings in 1 PR.

Part of 1) is also revisiting pandas optional dependencies. pandas has like 25+ optional dependencies and there has been discussion if we could also spin off some dependencies elsewhere (#45433).

A good first step as you mention would to be propose dependency groupings and gain some feedback.

[JMBurley]

I agree with the overall path to solving all extras_require options, and that that would be a nice improvement to pandas.

However I do want to highlight that that will be a quite large discussion that will inevitably reveal a series of packaging problems and lead to this issue dragging on and taking a while to resolve.

---

s3fs example

s3fs is required for s3 interaction but (because s3fs>=0.5.0 requires aiobotocore) that is mutually incompatible with moto[boto3] (some unresolved issues explaining the aiobotocore botocore problem).

It isn't actively documented in pandas, but right now the entire stability of pandas is pinned to two s3fs releases from March 2020 because pandas needs moto for tests, and moto cannot coexist with s3fs>0.5.0 and pandas requires s3fs>0.4.0.

But pandas has non-professional users who don't care about writing tests (ie. don't care about moto) and therefore might prefer pandas to allow a more recent version of s3fs.

So what versions of s3fs would we pin pandas to?

---

I'm diving deep into that one s3fs issue to flag that solving packaging in full is not going to be quick, and the fundamental work to understand packaging and versioning is not complete (eg. environment.yml has required indirectly may be? as a comment, and many dependencies have no stated versions although other requirements / setup files and documentation may suggest versions). There will be multiple issues like s3fs.

I propose incremental improvement. Allowing the recommended installs to be managed dependencies is an easy improvement to pandas packaging + dependencies that we can make right now that will save developers some serious headaches. How we solve that will not change in future and it is independent of the optional dependencies that extend functionality. Perhaps we can bundle that under an optional extra performance or recommended?

We can then work towards the larger, laudable goal of allowing functionality groupings (eg. s3, gcp, excel, parquet, sql, html, plots, dask) as extras.

Thoughts?

Groupings

(Based off current functionality enhancements in setup & environment.yml)

extras_require name	Enforces
recommended:	numexper>=2.7.1, bottleneck>=1.3.1
s3:	TBC
gcp:	TBC
excel:	TBC
parquet:	TBC
sql:	TBC
html:	TBC
plots:	TBC
dask:	TBC

[bluss]

I would wish for that pandas not recommend bottleneck until there is a mitigation for #42878

[JMBurley]

@mroeschke pinging on the above. I would still advocate that the best path is to package what can be trivialise packaged right now in order to improve pandas as-is, and then finalise more difficult packaging decisions later.

Otherwise, I think the packaging below would be in line with current understanding.

Groupings

(Based off current functionality enhancements in setup & environment.yml, install_guide where available, or open-ended >=current_stable_version when no information available regarding pandas compatibility)

extras_require name	Enforces
recommended:	numexper>=2.7.1; bottleneck>=1.3.1
computation:	SciPy >=1.4.1; numba>=0.50.1 ; xarray>=0.15.1
s3:	s3fs>=0.4.0 ; boto3>=1.22.7
gcp:	gcsfs>=0.6.0 ; pandas-gbq>=0.14.0 ;
excel:	xlrd >=2.0.1 ; xlwt >=1.3.0 ; xlsxwriter v1.2.2 ; openpyxl >=3.0.3 ; pyxlsb >=1.0.6
parquet:	fastparquet>=0.4.0; pyarrow>=1.0.1
feather:	pyarrow>=1.0.1
hdf5:	PyTables>=3.6.1 ; blosc>=1.20.1 ;
sql:	SQLAlchemy>=1.4.0 ; psycopg2>=2.8.4 ; pymysql>=0.10.1
html:	BeautifulSoup4>=4.8.2 ; html5lib>=1.1 ; lxml>=4.5.0
viz:	matplotlib>=3.3.2 ; Jinja2>=2.11 ; tabulate>=0.8.7

Notes:

• recommended: It seems that Numba should be in this list for performance boost on rolling operations?
• s3: fsspec is explicit dependency of s3fs, pandas can ignore.
• hdf5: zlib not pip package, so ignore in setup. nb./ pandas install_guide should recommend >=1.1.4 due to security flaws in prior zlib
• clipboard: I am ignoring because OS-dependent packaging is outside of current scope, and such functionality very rarely relevant to dockerising production code (thus, low priority IMO)

I can incorporate all of the above in my PR to manage optional dependencies via [options.extras_require] if there is consensus that we should follow this path. Thoughts?

Note that there would be an open question on how to ensure that https://pandas.pydata.org/docs/getting_started/install.html is kept current with the actual packaging.

[mroeschke]

Sorry for the delay @JMBurley.

On the surface these groupings seem reasonable, I would however be weary to make a final call here without more buy in from the core team @pandas-dev/pandas-core

This type of enhancement may warrant going through our developing enhancement proposal process #47444 as this would be a larger change for the library, so let's wait for those developments before moving forward here.

[bashtage]

Would it make sense to have a kitchen sink option, e.g., pip install pandas[all]?

[bashtage]

• recommended: It seems that Numba should be in this list for performance boost on rolling operations?

I think numba can be a bit too fiddly on some OS/arch types to be "recommended".