arm64 macOS wheel is broken

[messense]

In [1]: from ceresdb_client import Client
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from ceresdb_client import Client

File ~/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/__init__.py:1, in <module>
----> 1 from .ceresdb_client import *
      3 __doc__ = ceresdb_client.__doc__
      4 if hasattr(ceresdb_client, "__all__"):

ImportError: dlopen(/Users/messense/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/ceresdb_client/ceresdb_client.abi3.so, 0x0002): symbol not found in flat namespace (_aes_hw_cbc_encrypt)

symbol not found in flat namespace (_aes_hw_cbc_encrypt) looks like the same kind of issue as ray-project/ray#14697, I think it might be caused by misconfiguration of grpc or boringssl when cross compiling from x86_64 macOS to arm64 macOS.

[messense]

BTW, boringssl-src dependency has problem cross compiling to Linux aarch64 too. It might be related to this.

error: failed to run custom build command for `grpcio-sys v0.9.1+1.38.0`

Caused by:
  process didn't exit successfully: `/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/release/build/grpcio-sys-f1fff01a9ba9e94e/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=grpc_wrap.cc
  cargo:rerun-if-changed=grpc
  cargo:rerun-if-env-changed=UPDATE_BIND
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=GRPCIO_SYS_USE_PKG_CONFIG
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CXX
  OPT_LEVEL = Some("3")
  TARGET = Some("aarch64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = Some("aarch64-unknown-linux-gnu-gcc")
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fp,neon,pmuv3")
  CMAKE_TOOLCHAIN_FILE_aarch64-unknown-linux-gnu = None
  CMAKE_TOOLCHAIN_FILE_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_TOOLCHAIN_FILE = None
  CMAKE_TOOLCHAIN_FILE = None
  CMAKE_GENERATOR_aarch64-unknown-linux-gnu = None
  CMAKE_GENERATOR_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_GENERATOR = None
  CMAKE_GENERATOR = None
  CMAKE_PREFIX_PATH_aarch64-unknown-linux-gnu = None
  CMAKE_PREFIX_PATH_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_PREFIX_PATH = None
  CMAKE_PREFIX_PATH = None
  CMAKE_aarch64-unknown-linux-gnu = None
  CMAKE_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE = None
  CMAKE = None
  running: "cmake" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl" "-DCMAKE_INSTALL_PREFIX=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_BUILD_TYPE=Release"
  -- The C compiler identification is GNU 6.5.0
  -- The CXX compiler identification is GNU 6.5.0
  -- Check for working C compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc
  -- Check for working C compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++
  -- Check for working CXX compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- The ASM compiler identification is GNU
  -- Found assembler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build/build
  running: "cmake" "--build" "." "--target" "ssl" "--config" "Release" "--parallel" "2"
  Scanning dependencies of target ssl
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_both.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/bio_ssl.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_lib.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_pkt.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_srtp.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/dtls_method.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/dtls_record.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/encrypted_client_hello.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handoff.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake_client.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake_server.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_both.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_lib.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_pkt.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_aead_ctx.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_asn1.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_buffer.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_cert.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_cipher.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_file.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_key_share.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_lib.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_privkey.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_session.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_stat.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_transcript.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_versions.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_x509.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/t1_enc.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/t1_lib.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_both.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_client.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_enc.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_server.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls_method.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls_record.cc.o
  [100%] Linking CXX static library libssl.a
  [100%] Built target ssl
  cargo:root=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build
  running: "cmake" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl" "-DCMAKE_INSTALL_PREFIX=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_BUILD_TYPE=Release"
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build/build
  running: "cmake" "--build" "." "--target" "crypto" "--config" "Release" "--parallel" "2"
  Scanning dependencies of target crypto
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/cipher_extra/aes128gcmsiv-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/chacha/chacha-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/aesni-gcm-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/cipher_extra/chacha20_poly1305_x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/ghash-ssse3-x86_64.S.o
  [  2%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/aesni-x86_64.S.o
  [  2%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/ghash-x86_64.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/md5-x86_64.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/p[256](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:262)-x86_64-asm.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/p256_beeu-x86_64-asm.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/rdrand-x86_64.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/rsaz-avx2.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha1-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha256-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha512-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/vpaes-x86_64.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/x86_64-mont.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/x86_64-mont5.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/test/trampoline-x86_64.S.o
  [  7%] Building C object CMakeFiles/crypto.dir/err_data.c.o
  [  8%] Building ASM object CMakeFiles/crypto.dir/src/crypto/hrss/asm/poly_rq_mul.S.o
  [  8%] Building C object CMakeFiles/crypto.dir/src/crypto/asn1/a_bitstr.c.o

  --- stderr
  make: warning: -j2 forced in submake: resetting jobserver mode.
  make: warning: -j2 forced in submake: resetting jobserver mode.
  /root/.cargo/registry/src/github.com-1ecc6[299](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:305)db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S: Assembler messages:
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[320](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:326): Error: unknown pseudo-op: `.att_syntax'
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[323](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:329): Error: unknown mnemonic `push' -- `push %rbp'
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[325](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:331): Error: bad register expression
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[326](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:332): Error: unknown mnemonic `movq' -- `movq %rsp,%rbp'

[messense]

I think rust-lang/cmake-rs#158 could resolve this, need to test it out.

[messense]

#21 changed the error to symbol not found in flat namespace (___isPlatformVersionAtLeast)

In [1]: from ceresdb_client import Client
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from ceresdb_client import Client

File ~/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/__init__.py:1, in <module>
----> 1 from .ceresdb_client import *
      3 __doc__ = ceresdb_client.__doc__
      4 if hasattr(ceresdb_client, "__all__"):

ImportError: dlopen(/Users/messense/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/ceresdb_client.abi3.so, 0x0002): symbol not found in flat namespace (___isPlatformVersionAtLeast)

[messense]

symbol not found in flat namespace (___isPlatformVersionAtLeast) looks like the same kind of issue as alexcrichton/curl-rust#279 which was fixed by linking against clang runtime. I think it might need to be addressed in boringssl-src or grpc-sys.

$ objdump --syms /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
...
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a(os_version_check.c.o):

SYMBOL TABLE:
0000000000000000 l     F __TEXT,__text ltmp0
0000000000000bc0 l     O __DATA,__bss __MergedGlobals
00000000000000e8 l     F __TEXT,__text _compatibilityInitializeAvailabilityCheck
00000000000004a0 l     F __TEXT,__text __initializeAvailabilityCheck
000000000000023c l     F __TEXT,__text _initializeAvailabilityCheck
00000000000007d0 l     F __TEXT,__text ___isPlatformOrVariantPlatformVersionAtLeast.cold.1
0000000000000868 l     O __TEXT,__cstring l_.str.3
0000000000000884 l     O __TEXT,__cstring l_.str.4
0000000000000895 l     O __TEXT,__cstring l_.str.5
00000000000008b1 l     O __TEXT,__cstring l_.str.6
00000000000008ce l     O __TEXT,__cstring l_.str.7
00000000000008ee l     O __TEXT,__cstring l_.str.8
000000000000090e l     O __TEXT,__cstring l_.str.9
0000000000000923 l     O __TEXT,__cstring l_.str.10
000000000000092f l     O __TEXT,__cstring l_.str.11
0000000000000941 l     O __TEXT,__cstring l_.str.12
0000000000000954 l     O __TEXT,__cstring l_.str.13
000000000000095e l     O __TEXT,__cstring l_.str.14
000000000000098f l     O __TEXT,__cstring l_.str.15
0000000000000991 l     O __TEXT,__cstring l_.str.16
00000000000009a0 l     O __TEXT,__cstring l_.str.17
00000000000007f4 l     O __TEXT,__cstring l___func__.__isPlatformOrVariantPlatformVersionAtLeast
0000000000000820 l     O __TEXT,__cstring l_.str.1
0000000000000833 l     O __TEXT,__cstring l_.str.2
00000000000007f4 l     O __TEXT,__cstring ltmp1
0000000000000bc0 l     O __DATA,__bss ltmp2
00000000000009b0 l     O __LD,__compact_unwind ltmp3
0000000000000a90 l     O __TEXT,__eh_frame ltmp4
0000000000000000 g     F __TEXT,__text ___isOSVersionAtLeast
0000000000000244 g     F __TEXT,__text ___isPlatformOrVariantPlatformVersionAtLeast
00000000000000f0 g     F __TEXT,__text ___isPlatformVersionAtLeast
0000000000000000         *UND* ___assert_rtn
0000000000000000         *UND* ___stack_chk_fail
0000000000000000         *UND* ___stack_chk_guard
0000000000000000  w      *UND* _dispatch_once_f
0000000000000000         *UND* _dlsym
0000000000000000         *UND* _fclose
0000000000000000         *UND* _fopen
0000000000000000         *UND* _fread
0000000000000000         *UND* _free
0000000000000000         *UND* _fseek
0000000000000000         *UND* _ftell
0000000000000000         *UND* _malloc
0000000000000000         *UND* _rewind
0000000000000000         *UND* _sscanf
...

[messense]

I wonder why didn't you use the pure Rust tonic crate for grpc support, grpc-rs build is quite complicated.

[waynexia]

I've wondered about this too. The key reason was performance. But we haven't compared them for a long time so the situation may have changed.

[messense]

@waynexia https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-04-23-bench-results

[waynexia]

@waynexia https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-04-23-bench-results

Nice link! Looks like grpcio is still faster than tonic in the multicore machine... I'm afraid we need to keep paying for it for a while

[messense]

I've confirmed that it works on M1 macOS with 9a417c4