QST: Python 3.9 testing in CI

[wumpus]

I use Travis CI to test, and I've tested all of my projects against the upcoming python 3.9 release. The only problem I'm seeing is dependent packages is pandas.

Pandas + 3.9 takes a long time to build and then dies with a compiler error. It also does this when I install cython first, which is the usual workaround when a new version of Python is close to release. Is some not-yet-mainstream cython version needed? I suppose there will be a wheel by the time 3.9.0 comes out in October?

Example CI log: https://travis-ci.com/github/wumpus/paramsurvey/jobs/383337409

  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: ‘PyUnicode_AsUnicode’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

The logfile contains complete version information about the travisci environment.

[jbrockmendel]

@WillAyd is cython3 the blocker here?

[WillAyd]

I think the problem is we import PyUnicode_GET_SIZE in writers.pyx which is deprecated. Might be able to swap out with PyUnicode_GET_LENGTH assuming that is available in Cython

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_GET_SIZE

[WillAyd]

@wumpus if you'd like to try I think you just need to update this and any occurrence within the module

pandas/pandas/_libs/writers.pyx

Line 5 in ddf2f05

from cpython.unicode cimport PyUnicode_GET_SIZE

[wumpus]

I made that one change, and (no surprise) there are other calls to the "legacy" Unicode interface beyond that first one.

The deprecation plan is PEP 623 and py3.9 has the first deprecations.

I'm confused, though, why this isn't already fixed in pandas -- is -Wall turned off the the pandas CI build under py3.9?

[WillAyd]

We have the warnings turned on but they must not be throwing a deprecation warning until 3.9.

Can you post the full error list?

[wumpus]

Well, with just the one change to change GET_SIZE to GET_LENGTH (https://travis-ci.com/github/wumpus/pandas-appender/jobs/384012415) this is the next error:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DNPY_NO_DEPRECATED_API=0 -I/tmp/pip-build-env-589femm7/overlay/lib/python3.9/site-packages/numpy/core/include -I/home/travis/virtualenv/python3.9-dev/include -I/opt/python/3.9-dev/include/python3.9 -c pandas/_libs/writers.c -o build/temp.linux-x86_64-3.9/pandas/_libs/writers.o -Werror
  pandas/_libs/writers.c: In function ‘__pyx_f_6pandas_5_libs_7writers_word_len’:
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: ‘PyUnicode_AsUnicode’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

PEP 623 appears to have a complete list of what was deprecated in py3.9. And yes, they were first deprecated in 3.9.

Edit: I looked at the Pandas travis CI looking for a successful 3.9 build, got tired of looking before I found one.

[WillAyd]

Strange - I made the edits above and it seemed to get rid of those warnings. I'll push up a PR soon

[wumpus]

Well, I'm getting them with cython-0.29.21 (the current release), and the symbols aren't in pandas, they're inserted by cython.

[wumpus]

@WillAyd Looks like your PR built in Travis with cython-3, which is what @jbrockmendel asked you up top. With that clue, now I can try it for myself.