importing pandas breaks after successfully installing pandas with pip

[rvernica]

Code Sample, a copy-pastable example if possible

> docker run --tty --interactive --rm python:2.7.12-alpine ash
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
v3.4.6-150-gf952adf [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.6-83-g67e50bc [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5983 distinct packages available
/ # apk add cython g++ musl-dev
(1/21) Upgrading musl (1.1.14-r12 -> 1.1.14-r14)
(2/21) Installing libffi (3.2.1-r2)
(3/21) Installing python (2.7.12-r0)
(4/21) Installing cython (0.24-r1)
(5/21) Installing libgcc (5.3.0-r0)
(6/21) Installing libstdc++ (5.3.0-r0)
(7/21) Installing binutils-libs (2.26-r1)
(8/21) Installing binutils (2.26-r1)
(9/21) Installing gmp (6.1.0-r0)
(10/21) Installing isl (0.14.1-r0)
(11/21) Installing libgomp (5.3.0-r0)
(12/21) Installing libatomic (5.3.0-r0)
(13/21) Installing pkgconf (0.9.12-r0)
(14/21) Installing pkgconfig (0.25-r1)
(15/21) Installing mpfr3 (3.1.2-r0)
(16/21) Installing mpc1 (1.0.3-r0)
(17/21) Installing gcc (5.3.0-r0)
(18/21) Installing musl-dev (1.1.14-r14)
(19/21) Installing libc-dev (0.7-r0)
(20/21) Installing g++ (5.3.0-r0)
(21/21) Upgrading musl-utils (1.1.14-r12 -> 1.1.14-r14)
Executing busybox-1.24.2-r11.trigger
OK: 230 MiB in 51 packages
/ # ln -s locale.h /usr/include/xlocale.h
/ # pip install numpy==1.11.3 pandas==0.20.2
Collecting numpy==1.11.3
  Downloading numpy-1.11.3.zip (4.7MB)
    100% |████████████████████████████████| 4.7MB 242kB/s 
Collecting pandas==0.20.2
  Downloading pandas-0.20.2.tar.gz (10.3MB)
    100% |████████████████████████████████| 10.3MB 124kB/s 
Collecting python-dateutil (from pandas==0.20.2)
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 994kB/s 
Collecting pytz>=2011k (from pandas==0.20.2)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |████████████████████████████████| 491kB 874kB/s 
Collecting six>=1.5 (from python-dateutil->pandas==0.20.2)
  Downloading six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: numpy, pandas
  Running setup.py bdist_wheel for numpy ... done
  Stored in directory: /root/.cache/pip/wheels/8f/d6/97/4cc26777ee05997164e480e2e5de2c9e9c4e32dfe325ea5bd6
  Running setup.py bdist_wheel for pandas ... done
  Stored in directory: /root/.cache/pip/wheels/fc/fd/0a/5857e8bee7d5cc3ca22228fb7ab5efa43be57c0f9103e54098
Successfully built numpy pandas
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.11.3 pandas-0.20.2 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ # python -c "import pandas"
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/pandas/__init__.py", line 35, in <module>
    "the C extensions first.".format(module))
ImportError: C extension: umpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

Problem description

Installing Pandas with pip is successful, but import pandas breaks.

If I issue two separate pip install commands, one for numpy and one for pandas, it works:

> docker run --tty --interactive --rm python:2.7.12-alpine ash
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
v3.4.6-150-gf952adf [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.6-83-g67e50bc [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5983 distinct packages available
/ # apk add cython g++ musl-dev
(1/21) Upgrading musl (1.1.14-r12 -> 1.1.14-r14)
(2/21) Installing libffi (3.2.1-r2)
(3/21) Installing python (2.7.12-r0)
(4/21) Installing cython (0.24-r1)
(5/21) Installing libgcc (5.3.0-r0)
(6/21) Installing libstdc++ (5.3.0-r0)
(7/21) Installing binutils-libs (2.26-r1)
(8/21) Installing binutils (2.26-r1)
(9/21) Installing gmp (6.1.0-r0)
(10/21) Installing isl (0.14.1-r0)
(11/21) Installing libgomp (5.3.0-r0)
(12/21) Installing libatomic (5.3.0-r0)
(13/21) Installing pkgconf (0.9.12-r0)
(14/21) Installing pkgconfig (0.25-r1)
(15/21) Installing mpfr3 (3.1.2-r0)
(16/21) Installing mpc1 (1.0.3-r0)
(17/21) Installing gcc (5.3.0-r0)
(18/21) Installing musl-dev (1.1.14-r14)
(19/21) Installing libc-dev (0.7-r0)
(20/21) Installing g++ (5.3.0-r0)
(21/21) Upgrading musl-utils (1.1.14-r12 -> 1.1.14-r14)
Executing busybox-1.24.2-r11.trigger
OK: 230 MiB in 51 packages
/ # ln -s locale.h /usr/include/xlocale.h
/ # pip install numpy==1.11.3 
Collecting numpy==1.11.3
  Downloading numpy-1.11.3.zip (4.7MB)
    100% |████████████████████████████████| 4.7MB 221kB/s 
Building wheels for collected packages: numpy
  Running setup.py bdist_wheel for numpy ... done
  Stored in directory: /root/.cache/pip/wheels/8f/d6/97/4cc26777ee05997164e480e2e5de2c9e9c4e32dfe325ea5bd6
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.11.3
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ # pip install pandas==0.20.2
Collecting pandas==0.20.2
  Downloading pandas-0.20.2.tar.gz (10.3MB)
    100% |████████████████████████████████| 10.3MB 125kB/s 
Collecting python-dateutil (from pandas==0.20.2)
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 787kB/s 
Collecting pytz>=2011k (from pandas==0.20.2)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |████████████████████████████████| 491kB 1.0MB/s 
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in /usr/local/lib/python2.7/site-packages (from pandas==0.20.2)
Collecting six>=1.5 (from python-dateutil->pandas==0.20.2)
  Downloading six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: pandas
  Running setup.py bdist_wheel for pandas ... done
  Stored in directory: /root/.cache/pip/wheels/fc/fd/0a/5857e8bee7d5cc3ca22228fb7ab5efa43be57c0f9103e54098
Successfully built pandas
Installing collected packages: six, python-dateutil, pytz, pandas
Successfully installed pandas-0.20.2 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ # python -c "import pandas"

[TomAugspurger]

Looks pretty similar to #11383 (comment)

Can you try with verbose mode to see what version of numpy it pip grabs when you do them at the same time?

[rvernica]

It looks like it is picking up numpy 1.13.0:

gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include -I/usr/local/include/python2.7 -c pandas/_libs/testing.c -o build/temp.linux-x86_64-2.7/pandas/_libs/testing.o -Wno-unused-function
  gcc -shared build/temp.linux-x86_64-2.7/pandas/_libs/testing.o -L/usr/local/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/pandas/_libs/testing.so
  building 'pandas._libs.lib' extension
  gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include -I/usr/local/include/python2.7 -c pandas/_libs/lib.c -o build/temp.linux-x86_64-2.7/pandas/_libs/lib.o -Wno-unused-function
  In file included from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                   from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:18,
                   from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/arrayobject.h:4,
                   from pandas/_libs/lib.c:435:
  /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

[TomAugspurger]

Maybe try reporting on the pip issue tracker then. It's possible there's something wrong with our setup.py but a bug pip or setuptools seems more likely.

[jorisvandenbossche]

@rvernica is there a specific reason that you are building pandas from source, instead of downloading and installing the wheel? Because with the wheel, it should install without problems I think.

[TomAugspurger]

Ah I didn't see that the pip version was that old. It may already be fixed then.

[rvernica]

@jorisvandenbossche I am just doing pip install, pip chooses to build it from source probably because there is no wheel.

I tried upgrading pip first:

/ # pip install --upgrade pip
Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 480kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-9.0.1

But the result is the same, pip succeeds but import pandas fails.

[jorisvandenbossche]

There should certainly be a wheel for pandas-0.20.2, so not sure why it is not downloading it ..

[jorisvandenbossche]

However, what system is it? It is maybe not compatible with manylinux requirements.

But it is indeed the same issue as in #11383. So when doing install from source, installing with pip breaks if you try to install numpy and pandas in one go (and if you are not installing the latest version of numpy). The reason is that pip seems to build pandas in a separate env for which it uses the latest numpy, but in the actual env an older version of numpy is installed, and hence the errors. Solution is just do it in two steps (then numpy is already installed, and when building pandas it will use that and not fetch another one).
So either pip is just broken for this use case, or we do something wrong in our setup.py (but no expert in that)

[rvernica]

Makes sense. The system is Alpine Linux 3.4.

[TomAugspurger]

@rvernica did you report this on the pip issue tracker? I assumed I would be able to reproduce this with the --no-binary option, but that pulled down the correct numpy version.

[rvernica]

No, I didn't get the chance to report it.

[jorisvandenbossche]

I assumed I would be able to reproduce this with the --no-binary option, but that pulled down the correct numpy version.

@TomAugspurger did you specify an older version of numpy? (as that is needed to trigger it, eg pip install numpy==1.11 pandas==0.20)

[TomAugspurger]

Yeah I did pip install numpy==1.11.3 pandas==0.20.2 --no-binary=':all:' --verbose into a clean environment.