CI: Incompatible versions in resolved dependencies

[attack68]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

.

Issue Description

I tried to create a new development environment today.

Resolving the dependencies using pip install -r requirements-dev.txt took two hours before I cancelled it.

I then tried to use pip-compile from pip install pip-tools and I received the message:

There are incompatible versions in the resolved dependencies:
  botocore>=1.11 (from -r requirements-dev.in (line 35))
  botocore>=1.12.201 (from moto==2.2.8->-r requirements-dev.in (line 37))
  botocore<1.22.0,>=1.21.51 (from boto3==1.18.51->-r requirements-dev.in (line 34))
  botocore<1.20.107,>=1.20.106 (from aiobotocore==1.4.1->-r requirements-dev.in (line 72))

Having pinned boto3>=1.17.106,<1.17.107 since that is what aiobotocore packages with, the next error was:

There are incompatible versions in the resolved dependencies:
  fsspec<2021.6.0,>=0.7.4 (from -r requirements-dev.in (line 73))
  fsspec==2021.09.0 (from gcsfs==2021.9.0->-r requirements-dev.in (line 74))
  fsspec==2021.09.0 (from s3fs==2021.9.0->-r requirements-dev.in (line 71))
  fsspec (from fastparquet==0.7.1->-r requirements-dev.in (line 67))
  fsspec>=0.6.0 (from dask==2021.9.1->-r requirements-dev.in (line 26))

So then I pinned gcsfs==2021.5.0 and s3f2==2021.5.0.

It then compiled and I was able to install packages.

(adding a third complaint: new pyright checks need numpy >1.21.0 but numba needs less than that)

Expected Behavior

consistent deps.

Installed Versions

master as of 29-sep-21. python 3.8, mac os.

[lithomas1]

I have fixed the fsspec issue, and unpinned fsspec. I don't use pip to install my packages so not sure about the first error. Can you please provide us your pip version?

[attack68]

@lithomas1 using pip 21.2.4 in python3.9. It seems the fsspec issue is resolved.
boto3 still gives issues when resolving via pip-tools.

FYI the solved dependencies that I installed were:

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile requirements-dev.in
#
aiobotocore==1.4.2
    # via
    #   -r requirements-dev.in
    #   s3fs
aiohttp==3.7.4.post0
    # via
    #   aiobotocore
    #   gcsfs
aioitertools==0.8.0
    # via aiobotocore
alabaster==0.7.12
    # via sphinx
appdirs==1.4.4
    # via black
appnope==0.1.2
    # via
    #   ipykernel
    #   ipython
argon2-cffi==21.1.0
    # via notebook
asv==0.4.2
    # via -r requirements-dev.in
async-timeout==3.0.1
    # via aiohttp
attrs==21.2.0
    # via
    #   aiohttp
    #   flake8-bugbear
    #   hypothesis
    #   jsonschema
    #   pytest
babel==2.9.1
    # via sphinx
backcall==0.2.0
    # via ipython
backports.entry-points-selectable==1.1.0
    # via virtualenv
beautifulsoup4==4.10.0
    # via
    #   -r requirements-dev.in
    #   pydata-sphinx-theme
black==21.5b2
    # via -r requirements-dev.in
bleach==4.1.0
    # via nbconvert
blosc==1.10.6
    # via -r requirements-dev.in
boto3==1.17.106
    # via
    #   -r requirements-dev.in
    #   moto
botocore==1.20.106
    # via
    #   -r requirements-dev.in
    #   aiobotocore
    #   boto3
    #   moto
    #   s3transfer
bottleneck==1.3.2
    # via -r requirements-dev.in
cachetools==4.2.4
    # via google-auth
certifi==2021.10.8
    # via requests
cffi==1.14.6
    # via
    #   argon2-cffi
    #   cryptography
cfgv==3.3.1
    # via pre-commit
cftime==1.5.1
    # via -r requirements-dev.in
chardet==4.0.0
    # via aiohttp
charset-normalizer==2.0.6
    # via requests
click==8.0.3
    # via
    #   black
    #   flask
cloudpickle==2.0.0
    # via
    #   -r requirements-dev.in
    #   dask
coverage[toml]==6.0.1
    # via pytest-cov
cpplint==1.5.5
    # via -r requirements-dev.in
cramjam==2.4.0
    # via fastparquet
cryptography==35.0.0
    # via moto
cycler==0.10.0
    # via matplotlib
cython==0.29.24
    # via -r requirements-dev.in
dask==2021.9.1
    # via -r requirements-dev.in
debugpy==1.5.0
    # via ipykernel
decorator==5.1.0
    # via
    #   gcsfs
    #   ipython
defusedxml==0.7.1
    # via
    #   nbconvert
    #   odfpy
distlib==0.3.3
    # via virtualenv
docutils==0.17.1
    # via
    #   nbsphinx
    #   pydata-sphinx-theme
    #   sphinx
    #   sphinx-panels
entrypoints==0.3
    # via
    #   jupyter-client
    #   nbconvert
et-xmlfile==1.1.0
    # via openpyxl
execnet==1.9.0
    # via pytest-xdist
fastparquet==0.7.1
    # via -r requirements-dev.in
feedparser==6.0.8
    # via -r requirements-dev.in
filelock==3.3.0
    # via virtualenv
flake8==3.9.2
    # via
    #   -r requirements-dev.in
    #   flake8-bugbear
    #   flake8-comprehensions
    #   pandas-dev-flaker
flake8-bugbear==21.3.2
    # via -r requirements-dev.in
flake8-comprehensions==3.1.0
    # via -r requirements-dev.in
flask==2.0.2
    # via -r requirements-dev.in
fsspec==2021.10.0
    # via
    #   -r requirements-dev.in
    #   dask
    #   fastparquet
    #   gcsfs
    #   s3fs
gcsfs==2021.10.0
    # via -r requirements-dev.in
gitdb==4.0.7
    # via
    #   -r requirements-dev.in
    #   gitpython
gitpython==3.1.24
    # via -r requirements-dev.in
google-auth==2.3.0
    # via
    #   gcsfs
    #   google-auth-oauthlib
google-auth-oauthlib==0.4.6
    # via gcsfs
greenlet==1.1.2
    # via sqlalchemy
html5lib==1.1
    # via -r requirements-dev.in
hypothesis==6.23.2
    # via -r requirements-dev.in
identify==2.3.0
    # via pre-commit
idna==3.2
    # via
    #   requests
    #   yarl
imagesize==1.2.0
    # via sphinx
iniconfig==1.1.1
    # via pytest
ipykernel==6.4.1
    # via
    #   -r requirements-dev.in
    #   ipywidgets
    #   notebook
ipython==7.28.0
    # via
    #   -r requirements-dev.in
    #   ipykernel
    #   ipywidgets
ipython-genutils==0.2.0
    # via
    #   ipykernel
    #   ipywidgets
    #   nbformat
    #   notebook
ipywidgets==7.6.5
    # via -r requirements-dev.in
isort==5.9.3
    # via -r requirements-dev.in
itsdangerous==2.0.1
    # via flask
jedi==0.18.0
    # via ipython
jinja2==3.0.2
    # via
    #   -r requirements-dev.in
    #   flask
    #   moto
    #   nbconvert
    #   nbsphinx
    #   notebook
    #   numpydoc
    #   sphinx
jmespath==0.10.0
    # via
    #   boto3
    #   botocore
jsonschema==4.1.0
    # via nbformat
jupyter-client==7.0.6
    # via
    #   ipykernel
    #   nbclient
    #   notebook
jupyter-core==4.8.1
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
    #   notebook
jupyterlab-pygments==0.1.2
    # via nbconvert
jupyterlab-widgets==1.0.2
    # via ipywidgets
kiwisolver==1.3.2
    # via matplotlib
llvmlite==0.37.0
    # via numba
locket==0.2.1
    # via partd
lxml==4.6.3
    # via -r requirements-dev.in
markdown==3.3.4
    # via -r requirements-dev.in
markupsafe==2.0.1
    # via
    #   jinja2
    #   moto
matplotlib==3.4.3
    # via
    #   -r requirements-dev.in
    #   seaborn
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mccabe==0.6.1
    # via flake8
mistune==0.8.4
    # via nbconvert
more-itertools==8.10.0
    # via moto
moto==2.2.9
    # via -r requirements-dev.in
multidict==5.2.0
    # via
    #   aiohttp
    #   yarl
mypy==0.910
    # via -r requirements-dev.in
mypy-extensions==0.4.3
    # via
    #   black
    #   mypy
natsort==7.1.1
    # via -r requirements-dev.in
nbclient==0.5.4
    # via nbconvert
nbconvert==6.2.0
    # via
    #   -r requirements-dev.in
    #   nbsphinx
    #   notebook
nbformat==5.1.3
    # via
    #   -r requirements-dev.in
    #   ipywidgets
    #   nbclient
    #   nbconvert
    #   nbsphinx
    #   notebook
nbsphinx==0.8.7
    # via -r requirements-dev.in
nest-asyncio==1.5.1
    # via
    #   jupyter-client
    #   nbclient
nodeenv==1.6.0
    # via pre-commit
notebook==6.4.4
    # via
    #   -r requirements-dev.in
    #   widgetsnbextension
numba==0.54.1
    # via -r requirements-dev.in
numexpr==2.7.3
    # via -r requirements-dev.in
numpy==1.20.3
    # via
    #   -r requirements-dev.in
    #   bottleneck
    #   cftime
    #   fastparquet
    #   matplotlib
    #   numba
    #   numexpr
    #   pandas
    #   patsy
    #   pyarrow
    #   scipy
    #   seaborn
    #   statsmodels
    #   xarray
numpydoc==1.1.0
    # via -r requirements-dev.in
oauthlib==3.1.1
    # via requests-oauthlib
odfpy==1.4.1
    # via -r requirements-dev.in
openpyxl==3.0.9
    # via -r requirements-dev.in
packaging==21.0
    # via
    #   bleach
    #   dask
    #   pytest
    #   sphinx
pandas==1.3.3
    # via
    #   fastparquet
    #   pyreadstat
    #   seaborn
    #   statsmodels
    #   xarray
pandas-dev-flaker==0.2.0
    # via -r requirements-dev.in
pandoc==1.1.0
    # via -r requirements-dev.in
pandocfilters==1.5.0
    # via nbconvert
parso==0.8.2
    # via jedi
partd==1.2.0
    # via
    #   -r requirements-dev.in
    #   dask
pathspec==0.9.0
    # via black
patsy==0.5.2
    # via statsmodels
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
pillow==8.3.2
    # via matplotlib
platformdirs==2.4.0
    # via virtualenv
pluggy==1.0.0
    # via pytest
plumbum==1.7.0
    # via pandoc
ply==3.11
    # via pandoc
pre-commit==2.15.0
    # via -r requirements-dev.in
prometheus-client==0.11.0
    # via notebook
prompt-toolkit==3.0.20
    # via ipython
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
py==1.10.0
    # via
    #   pytest
    #   pytest-forked
pyarrow==5.0.0
    # via -r requirements-dev.in
pyasn1==0.4.8
    # via
    #   pyasn1-modules
    #   rsa
pyasn1-modules==0.2.8
    # via google-auth
pycodestyle==2.7.0
    # via
    #   -r requirements-dev.in
    #   flake8
pycparser==2.20
    # via cffi
pydata-sphinx-theme @ git+https://github.com/pydata/pydata-sphinx-theme.git@master
    # via -r requirements-dev.in
pyflakes==2.3.1
    # via flake8
pygments==2.10.0
    # via
    #   ipython
    #   jupyterlab-pygments
    #   nbconvert
    #   sphinx
pyparsing==2.4.7
    # via
    #   matplotlib
    #   packaging
pyreadstat==1.1.3
    # via -r requirements-dev.in
pyrsistent==0.18.0
    # via jsonschema
pytest==6.2.5
    # via
    #   -r requirements-dev.in
    #   pytest-asyncio
    #   pytest-cov
    #   pytest-cython
    #   pytest-forked
    #   pytest-instafail
    #   pytest-xdist
pytest-asyncio==0.15.1
    # via -r requirements-dev.in
pytest-cov==3.0.0
    # via -r requirements-dev.in
pytest-cython==0.1.1.post0
    # via -r requirements-dev.in
pytest-forked==1.3.0
    # via pytest-xdist
pytest-instafail==0.4.2
    # via -r requirements-dev.in
pytest-xdist==2.4.0
    # via -r requirements-dev.in
python-dateutil==2.8.2
    # via
    #   -r requirements-dev.in
    #   botocore
    #   jupyter-client
    #   matplotlib
    #   moto
    #   pandas
python-snappy==0.6.0
    # via -r requirements-dev.in
pytz==2021.3
    # via
    #   -r requirements-dev.in
    #   babel
    #   moto
    #   pandas
pyupgrade==2.29.0
    # via -r requirements-dev.in
pyyaml==5.4.1
    # via
    #   -r requirements-dev.in
    #   dask
    #   pre-commit
pyzmq==22.3.0
    # via
    #   jupyter-client
    #   notebook
regex==2021.10.8
    # via black
requests==2.26.0
    # via
    #   -r requirements-dev.in
    #   gcsfs
    #   moto
    #   requests-oauthlib
    #   responses
    #   sphinx
requests-oauthlib==1.3.0
    # via google-auth-oauthlib
responses==0.14.0
    # via moto
rsa==4.7.2
    # via google-auth
s3fs==2021.10.0
    # via -r requirements-dev.in
s3transfer==0.4.2
    # via boto3
scipy==1.7.1
    # via
    #   -r requirements-dev.in
    #   seaborn
    #   statsmodels
seaborn==0.11.2
    # via -r requirements-dev.in
send2trash==1.8.0
    # via notebook
sgmllib3k==1.0.0
    # via feedparser
six==1.16.0
    # via
    #   asv
    #   bleach
    #   cycler
    #   html5lib
    #   patsy
    #   python-dateutil
    #   responses
    #   thrift
    #   virtualenv
smmap==4.0.0
    # via gitdb
snowballstemmer==2.1.0
    # via sphinx
sortedcontainers==2.4.0
    # via hypothesis
soupsieve==2.2.1
    # via beautifulsoup4
sphinx==4.2.0
    # via
    #   -r requirements-dev.in
    #   nbsphinx
    #   numpydoc
    #   pydata-sphinx-theme
    #   sphinx-panels
sphinx-panels==0.6.0
    # via -r requirements-dev.in
sphinxcontrib-applehelp==1.0.2
    # via sphinx
sphinxcontrib-devhelp==1.0.2
    # via sphinx
sphinxcontrib-htmlhelp==2.0.0
    # via sphinx
sphinxcontrib-jsmath==1.0.1
    # via sphinx
sphinxcontrib-qthelp==1.0.3
    # via sphinx
sphinxcontrib-serializinghtml==1.1.5
    # via sphinx
sqlalchemy==1.4.25
    # via -r requirements-dev.in
statsmodels==0.13.0
    # via -r requirements-dev.in
tabulate==0.8.9
    # via -r requirements-dev.in
terminado==0.12.1
    # via notebook
testpath==0.5.0
    # via nbconvert
thrift==0.15.0
    # via fastparquet
tokenize-rt==4.1.0
    # via pyupgrade
toml==0.10.2
    # via
    #   black
    #   mypy
    #   pre-commit
    #   pytest
tomli==1.2.1
    # via coverage
toolz==0.11.1
    # via
    #   -r requirements-dev.in
    #   dask
    #   partd
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
    #   notebook
    #   terminado
traitlets==5.1.0
    # via
    #   ipykernel
    #   ipython
    #   ipywidgets
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
    #   nbsphinx
    #   notebook
types-pymysql==1.0.2
    # via -r requirements-dev.in
types-python-dateutil==2.8.0
    # via -r requirements-dev.in
types-pytz==2021.1.2
    # via -r requirements-dev.in
types-setuptools==57.4.0
    # via -r requirements-dev.in
typing-extensions==3.10.0.2
    # via
    #   aiohttp
    #   gitpython
    #   mypy
urllib3==1.26.7
    # via
    #   botocore
    #   requests
    #   responses
virtualenv==20.8.1
    # via pre-commit
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   html5lib
werkzeug==2.0.2
    # via
    #   flask
    #   moto
widgetsnbextension==3.5.1
    # via ipywidgets
wrapt==1.13.1
    # via aiobotocore
xarray==0.18.2
    # via -r requirements-dev.in
xlrd==2.0.1
    # via -r requirements-dev.in
xlsxwriter==3.0.1
    # via -r requirements-dev.in
xlwt==1.3.0
    # via -r requirements-dev.in
xmltodict==0.12.0
    # via moto
yarl==1.7.0
    # via aiohttp

# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools