BUILD: "error: incomplete definition of type 'struct _frame'" while compiling the datetime library on macOS & python 3.11

[burnpanck]

Installation check

• I have read the installation guide.

Platform

macOS-13.4.1-arm64-arm-64bit

Installation Method

Built from source

pandas Version

main (1c5c4ef)

Python Version

3.11.3

Installation Logs

The environment is a clean pyenv virtualenv with just pip install -r requirements-dev.txt applied,
(after CFLAGS="<setup include and linker directories for homebrew snappy>" pip install python-snappy).

(pandas-dev) ➜  pandas git:(main) python setup.py build_ext -j 4                         
/Users/yves/git-public/pandas/setup.py:19: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import parse_version
/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:66: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running build_ext
building 'pandas._libs.parsers' extension
building 'pandas._libs.lib' extension
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/lib.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/lib.o -Wno-error=unreachable-code
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/parsers.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/parsers.o -Wno-error=unreachable-code
building 'pandas._libs.ops_dispatch' extension
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/ops_dispatch.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/ops_dispatch.o -Wno-error=unreachable-code
building 'pandas._libs.pandas_datetime' extension
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/src/datetime/date_conversions.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/datetime/date_conversions.o -Wno-error=unreachable-code
pandas/_libs/ops_dispatch.c:3995:43: warning: 'ob_shash' is deprecated [-Wdeprecated-declarations]
            hash1 = ((PyBytesObject*)s1)->ob_shash;
                                          ^
/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11/cpython/bytesobject.h:7:5: note: 'ob_shash' has been explicitly marked deprecated here
    Py_DEPRECATED(3.11) Py_hash_t ob_shash;
    ^
/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11/pyport.h:336:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
pandas/_libs/ops_dispatch.c:3996:43: warning: 'ob_shash' is deprecated [-Wdeprecated-declarations]
            hash2 = ((PyBytesObject*)s2)->ob_shash;
                                          ^
/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11/cpython/bytesobject.h:7:5: note: 'ob_shash' has been explicitly marked deprecated here
    Py_DEPRECATED(3.11) Py_hash_t ob_shash;
    ^
/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11/pyport.h:336:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/src/datetime/pd_datetime.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/datetime/pd_datetime.o -Wno-error=unreachable-code
pandas/_libs/ops_dispatch.c:4665:5: error: incomplete definition of type 'struct _frame'
    __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/ops_dispatch.c:457:62: note: expanded from macro '__Pyx_PyFrame_SetLineNumber'
  #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
                                                      ~~~~~~~^
/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'struct _frame'
typedef struct _frame PyFrameObject;
               ^
2 warnings and 1 error generated.
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/src/vendored/numpy/datetime/np_datetime.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/vendored/numpy/datetime/np_datetime.o -Wno-error=unreachable-code
pandas/_libs/parsers.c:47389:21: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
                    CYTHON_FALLTHROUGH;
                    ^
pandas/_libs/parsers.c:382:34: note: expanded from macro 'CYTHON_FALLTHROUGH'
      #define CYTHON_FALLTHROUGH __attribute__((fallthrough))
                                 ^
pandas/_libs/parsers.c:47400:21: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
                    CYTHON_FALLTHROUGH;
                    ^
pandas/_libs/parsers.c:382:34: note: expanded from macro 'CYTHON_FALLTHROUGH'
      #define CYTHON_FALLTHROUGH __attribute__((fallthrough))
                                 ^
pandas/_libs/lib.c:68646:21: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
                    CYTHON_FALLTHROUGH;
                    ^
pandas/_libs/lib.c:378:34: note: expanded from macro 'CYTHON_FALLTHROUGH'
      #define CYTHON_FALLTHROUGH __attribute__((fallthrough))
                                 ^
pandas/_libs/lib.c:68657:21: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
                    CYTHON_FALLTHROUGH;
                    ^
pandas/_libs/lib.c:378:34: note: expanded from macro 'CYTHON_FALLTHROUGH'
      #define CYTHON_FALLTHROUGH __attribute__((fallthrough))
                                 ^
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/include -I/Users/yves/.pyenv/versions/3.11.3/envs/pandas-dev/include -I/Users/yves/.pyenv/versions/3.11.3/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c pandas/_libs/src/vendored/numpy/datetime/np_datetime_strings.c -o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/vendored/numpy/datetime/np_datetime_strings.o -Wno-error=unreachable-code
clang -bundle -undefined dynamic_lookup -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/datetime/date_conversions.o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/datetime/pd_datetime.o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/vendored/numpy/datetime/np_datetime.o build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/src/vendored/numpy/datetime/np_datetime_strings.o -o build/lib.macosx-13.3-arm64-cpython-311/pandas/_libs/pandas_datetime.cpython-311-darwin.so
2 warnings generated.
clang -bundle -undefined dynamic_lookup -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/parsers.o -o build/lib.macosx-13.3-arm64-cpython-311/pandas/_libs/parsers.cpython-311-darwin.so
2 warnings generated.
clang -bundle -undefined dynamic_lookup -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/readline/lib -L/Users/yves/.pyenv/versions/3.11.3/lib -L/opt/homebrew/lib -Wl,-rpath,/opt/homebrew/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib build/temp.macosx-13.3-arm64-cpython-311/pandas/_libs/lib.o -o build/lib.macosx-13.3-arm64-cpython-311/pandas/_libs/lib.cpython-311-darwin.so
error: command '/usr/bin/clang' failed with exit code 1

[burnpanck]

Actually, this issue here is fixed with Cython 3.0.0; so I guess this relates to #54497.

[lithomas1]

Can you follow the dev installation instructions?
https://pandas.pydata.org/docs/dev/development/contributing_environment.html#step-3-build-and-install-pandas

I noticed that you are still using the old setuptools build system (that will be removed with 2.1)

[burnpanck]

Ah right, my bad, I didn't pay attention to the version of documentation that I was following.
Indeed, with the dev instructions it seems to build successfully with Cython 0.29.36. So I guess this one is already solved on main, as long as people follow the instructions :-).

[lithomas1]

No worries, the docs are still stuck defaulting to the 2.0 instructions instead of the dev instructions, so it's totally not your fault.

(All will be fine once 2.1 comes out in next/next next week?)