Clang/LLVM TOT/12.x unable to build Python TOT

[llvmbot]

Bugzilla Link	49461
Version	trunk
OS	MacOS X
Attachments	Script to build clang and llvm, Script to build python
Reporter	LLVM Bugzilla Contributor
CC	@fhahn

Extended Description

Hi,

I am trying to build tot python from https://github.com/python/cpython using clang tot from: https://github.com/llvm/llvm-project

I am seeing the following error:

clang -dynamiclib -Wl,-single_module -flto -Wl,-export_dynamic -g -fprofile-instr-generate -undefined dynamic_lookup -Wl,-install_name,/Users/aditya/.local/Python3.10.0a6/lib/libpython3.10.dylib -Wl,-compatibility_version,3.10 -Wl,-current_version,3.10 -o libpython3.10.dylib Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/_abc.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/signalmodule.o Modules/_stat.o Modules/timemodule.o Modules/_threadmodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o -ldl -framework CoreFoundation -v

clang version 13.0.0 (https://github.com/adityaatluri/llvm-project.git ecdae5d)

Target: x86_64-apple-darwin20.3.0

Thread model: posix

InstalledDir: /Users/aditya/.local/llvm13_x86_64/bin

"/usr/bin/ld" /Users/aditya/.local/llvm13_x86_64/lib/clang/13.0.0/lib/darwin/libclang_rt.profile_osx.a -demangle -lto_library /Users/aditya/.local/llvm13_x86_64/lib/libLTO.dylib -dynamic -dylib -arch x86_64 -platform_version macos 11.0.0 11.1 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -undefined dynamic_lookup -undefined dynamic_lookup -o libpython3.10.dylib -single_module -export_dynamic -install_name /Users/aditya/.local/Python3.10.0a6/lib/libpython3.10.dylib -compatibility_version 3.10 -current_version 3.10 Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/_abc.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/signalmodule.o Modules/_stat.o Modules/timemodule.o Modules/_threadmodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o -ldl -framework CoreFoundation -sectalign __DATA __llvm_prf_cnts 0x4000 -sectalign __DATA __llvm_prf_data 0x4000 -lSystem /Users/aditya/.local/llvm13_x86_64/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a

Assertion failed: (It->second == &Insn && "Cached first special instruction is wrong!"), function validate, file /Users/aditya/Code/github/src/llvm-project/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp, line 87.

clang-13: error: unable to execute command: Abort trap: 6

clang-13: error: linker command failed due to signal (use -v to see invocation)

How I built clang:
Attached script (get_llvm_os_all.sh) that i used to build clang for different versions
I am using XCode 12.4 to build clang from source

How I built python:
Attached script (get_python.sh). I build openssl from source, if you already have it, remove configure flag for openssl

Platform:
MacOS big sur 11.2.2
Xcode: Version 12.4 (12D4e)
Xcode clang:
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
TOT clang:
clang version 13.0.0 (https://github.com/llvm/llvm-project.git ecdae5d)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Users/aditya/.local/llvm13_x86_64/bin

Let me know if you need more information

[fhahn]

Could you try to apply https://reviews.llvm.org/D98183, re-build libLTO (target is LTO) and run ld as above, but with -mllvm -save-temps as extra options?

This should contain a bunch of ld-temp.* files, which are the intermediary files for LTO. Could you check if opt -O3 crashes for the generated ld-temp.0.3.import.bc ? Or llc for ld-temp.0.5.precodegen.bc.

If any of those crashes, it would be much easier to reproduce if you could share the bitcode files directly.

[fhahn]

Could you try to apply https://reviews.llvm.org/D98183, re-build libLTO
(target is LTO) and run ld as above, but with -mllvm -save-temps as extra
options?

This should contain a bunch of ld-temp.* files, which are the intermediary
files for LTO. Could you check if opt -O3 crashes for the generated
ld-temp.0.3.import.bc ? Or llc for ld-temp.0.5.precodegen.bc.

If any of those crashes, it would be much easier to reproduce if you could
share the bitcode files directly.

You don't really need to apply the patch, you can also just pass -save-temps to ld without the patch and it should generate a *lto.bc. It would be great if you could check that against opt -O3 and the *lto.opt.bc against llc

[llvmbot]

Hi,
I did not understand "It would be great if you could check that against opt -O3 and the *lto.opt.bc against llc". Able to apply patch and generate ld-temp.0.3.import.bc file

[chsigg]

I fixed this change in c64a098. Please check if the fix is correct.