Skip to content

Commit ba11e1e

Browse files
asbasb
authored
[docs] Rewrite HowToCrossCompileLLVM (#129451)
The document has had a few minor tweaks over the years, but the last major piece of work on it was 2016, after first being introduced in 2013. My aim is to provide a clear and clean recipe for cross-compiling LLVM that: * Should be achievable for anyone on common variants of Linux (_including_ the step of acquiring a working sysroot). * I think I've kept the coverage of setting up acquiring a Debian sysroot minimal enough that it can reasonably be included. `debootstrap` is packaged for most common Linux distributions including non-Debian derived distributions like Arch Linux and Fedora. * Describes a setup that we can reasonably support within the community. * I realise with the ninja symlink canonicalisation issue I haven't completely avoided hacks, but I look particularly to point 2 under hacks in the current docs which talks about libraries on the host being found by CMake and adding `-L` and `-I` to try to hack around this. We've all been there and made these kind of temporary workarounds to see if we can get further, but it's very hard to support someone who has problems with a setup that's improperly leaking between the host and target like this. The approach I describe with a clean sysroot and setting appropriate `CMAKE_FIND_ROOT_PATH_MODE_*` settings doesn't have this issue. * Cuts down on extraneous / outdated information, especially where it is better covered elsewhere (e.g. detailed descriptions of CMake options not directly relevant to cross compilation). I've run through the instructions for AArch64, RISC-V (64-bit), and armhf.
1 parent 5686786 commit ba11e1e

File tree

1 file changed

+177
-154
lines changed

1 file changed

+177
-154
lines changed

‎llvm/docs/HowToCrossCompileLLVM.rst

Lines changed: 177 additions & 154 deletions
Original file line numberDiff line numberDiff line change
@@ -1,215 +1,238 @@
11
===================================================================
2-
How To Cross-Compile Clang/LLVM using Clang/LLVM
2+
How to cross-compile Clang/LLVM using Clang/LLVM
33
===================================================================
44

55
Introduction
6-
============
6+
------------
77

88
This document contains information about building LLVM and
9-
Clang on host machine, targeting another platform.
9+
Clang on a host machine, targeting another platform.
1010

1111
For more information on how to use Clang as a cross-compiler,
1212
please check https://clang.llvm.org/docs/CrossCompilation.html.
1313

14-
TODO: Add MIPS and other platforms to this document.
14+
This document describes cross-building a compiler in a single stage, using an
15+
existing ``clang`` install as the host compiler.
1516

16-
Cross-Compiling from x86_64 to ARM
17-
==================================
17+
.. note::
18+
These instructions have been tested for targeting 32-bit ARM, AArch64, or
19+
64-bit RISC-V from an x86_64 Linux host. But should be equally applicable to
20+
any other target.
1821

19-
In this use case, we'll be using CMake and Ninja, on a Debian-based Linux
20-
system, cross-compiling from an x86_64 host (most Intel and AMD chips
21-
nowadays) to a hard-float ARM target (most ARM targets nowadays).
22-
23-
The packages you'll need are:
24-
25-
* ``cmake``
26-
* ``ninja-build`` (from backports in Ubuntu)
27-
* ``gcc-4.7-arm-linux-gnueabihf``
28-
* ``gcc-4.7-multilib-arm-linux-gnueabihf``
29-
* ``binutils-arm-linux-gnueabihf``
30-
* ``libgcc1-armhf-cross``
31-
* ``libsfgcc1-armhf-cross``
32-
* ``libstdc++6-armhf-cross``
33-
* ``libstdc++6-4.7-dev-armhf-cross``
34-
35-
Configuring CMake
36-
-----------------
37-
38-
For more information on how to configure CMake for LLVM/Clang,
39-
see :doc:`CMake`.
40-
41-
The CMake options you need to add are:
42-
43-
* ``-DCMAKE_SYSTEM_NAME=<target-system>``
44-
* ``-DCMAKE_INSTALL_PREFIX=<install-dir>``
45-
* ``-DLLVM_HOST_TRIPLE=arm-linux-gnueabihf``
46-
* ``-DLLVM_TARGETS_TO_BUILD=ARM``
47-
48-
Note: ``CMAKE_CROSSCOMPILING`` is always set automatically when ``CMAKE_SYSTEM_NAME`` is set. Don't put ``-DCMAKE_CROSSCOMPILING=TRUE`` in your options.
49-
50-
Also note that ``LLVM_HOST_TRIPLE`` specifies the triple of the system
51-
that the cross built LLVM is going to run on - the flag is named based
52-
on the autoconf build/host/target nomenclature. (This flag implicitly sets
53-
other defaults, such as ``LLVM_DEFAULT_TARGET_TRIPLE``.)
22+
Setting up a sysroot
23+
--------------------
5424

55-
If you're compiling with GCC, you can use architecture options for your target,
56-
and the compiler driver will detect everything that it needs:
25+
You will need a sysroot that contains essential build dependencies compiled
26+
for the target architecture. In this case, we will be using CMake and Ninja on
27+
a Linux host and compiling against a Debian sysroot. Detailed instructions on
28+
producing sysroots are outside of the scope of this documentation, but the
29+
following instructions should work on any Linux distribution with these
30+
pre-requisites:
5731

58-
* ``-DCMAKE_CXX_FLAGS='-march=armv7-a -mcpu=cortex-a9 -mfloat-abi=hard'``
32+
* ``binfmt_misc`` configured to execute ``qemu-user`` for binaries of the
33+
target architecture. This is done by installing the ``qemu-user-static``
34+
and ``binfmt-support`` packages on Debian-derived distributions.
35+
* Root access (setups involving ``proot`` or other tools to avoid this
36+
requirement may be possible, but aren't described here).
37+
* The ``debootstrap`` tool. This is available in most distributions.
5938

60-
However, if you're using Clang, the driver might not be up-to-date with your
61-
specific Linux distribution, version or GCC layout, so you'll need to fudge.
39+
The following snippet will initialise sysroots for 32-bit Arm, AArch64, and
40+
64-bit RISC-V (just pick the target(s) you are interested in):
6241

63-
In addition to the ones above, you'll also need:
42+
.. code-block:: bash
6443
65-
* ``--target=arm-linux-gnueabihf`` or whatever is the triple of your cross GCC.
66-
* ``'--sysroot=/usr/arm-linux-gnueabihf'``, ``'--sysroot=/opt/gcc/arm-linux-gnueabihf'``
67-
or whatever is the location of your GCC's sysroot (where /lib, /bin etc are).
68-
* Appropriate use of ``-I`` and ``-L``, depending on how the cross GCC is installed,
69-
and where are the libraries and headers.
44+
sudo debootstrap --arch=armhf --variant=minbase --include=build-essential,symlinks stable sysroot-deb-armhf-stable
45+
sudo debootstrap --arch=arm64 --variant=minbase --include=build-essential,symlinks stable sysroot-deb-arm64-stable
46+
sudo debootstrap --arch=riscv64 --variant=minbase --include=build-essential,symlinks unstable sysroot-deb-riscv64-unstable
7047
71-
You may also want to set the ``LLVM_NATIVE_TOOL_DIR`` option - pointing
72-
at a directory with prebuilt LLVM tools (``llvm-tblgen``, ``clang-tblgen``
73-
etc) for the build host, allowing you to them reuse them if available.
74-
E.g. ``-DLLVM_NATIVE_TOOL_DIR=<path-to-native-llvm-build>/bin``.
75-
If the option isn't set (or the directory doesn't contain all needed tools),
76-
the LLVM cross build will automatically launch a nested build to build the
77-
tools that are required.
48+
The created sysroot may contain absolute symlinks, which will resolve to a
49+
location within the host when accessed during compilation, so we must convert
50+
any absolute symlinks to relative ones:
7851

79-
The CXX flags define the target, cpu (which in this case
80-
defaults to ``fpu=VFP3`` with NEON), and forcing the hard-float ABI. If you're
81-
using Clang as a cross-compiler, you will *also* have to set ``--sysroot``
82-
to make sure it picks the correct linker.
52+
.. code-block:: bash
8353
84-
When using Clang, it's important that you choose the triple to be *identical*
85-
to the GCC triple and the sysroot. This will make it easier for Clang to
86-
find the correct tools and include headers. But that won't mean all headers and
87-
libraries will be found. You'll still need to use ``-I`` and ``-L`` to locate
88-
those extra ones, depending on your distribution.
54+
sudo chroot sysroot-of-your-choice symlinks -cr .
8955
90-
Most of the time, what you want is to have a native compiler to the
91-
platform itself, but not others. So there's rarely a point in compiling
92-
all back-ends. For that reason, you should also set the
93-
``TARGETS_TO_BUILD`` to only build the back-end you're targeting to.
9456
95-
You must set the ``CMAKE_INSTALL_PREFIX``, otherwise a ``ninja install``
96-
will copy ARM binaries to your root filesystem, which is not what you
97-
want.
57+
Configuring CMake and building
58+
------------------------------
9859

99-
Hacks
100-
-----
60+
For more information on how to configure CMake for LLVM/Clang,
61+
see :doc:`CMake`. Following CMake's recommended practice, we will create a
62+
`toolchain file
63+
<https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html#toolchain-files>`_.
10164

102-
There are some bugs in current LLVM, which require some fiddling before
103-
running CMake:
65+
The following assumes you have a system install of ``clang`` and ``lld`` that
66+
will be used for cross compiling and that the listed commands are executed
67+
from within the root of a checkout of the ``llvm-project`` git repository.
10468

105-
#. If you're using Clang as the cross-compiler, there is a problem in
106-
the LLVM ARM back-end that is producing absolute relocations on
107-
position-independent code (``R_ARM_THM_MOVW_ABS_NC``), so for now, you
108-
should disable PIC:
69+
First, set variables in your shell session that will be used throughout the
70+
build instructions:
10971

11072
.. code-block:: bash
11173
112-
-DLLVM_ENABLE_PIC=False
74+
SYSROOT=$HOME/sysroot-deb-arm64-stable
75+
TARGET=aarch64-linux-gnu
76+
CFLAGS=""
11377
114-
This is not a problem, since Clang/LLVM libraries are statically
115-
linked anyway, it shouldn't affect much.
78+
To customise details of the compilation target or choose a different
79+
architecture altogether, change the ``SYSROOT``,
80+
``TARGET``, and ``CFLAGS`` variables to something matching your target. For
81+
example, for 64-bit RISC-V you might set
82+
``SYSROOT=$HOME/sysroot-deb-riscv64-unstable``, ``TARGET=riscv64-linux-gnu``
83+
and ``CFLAGS="-march=rva20u64"``. Refer to documentation such as your target's
84+
compiler documentation or processor manual for guidance on which ``CFLAGS``
85+
settings may be appropriate. The specified ``TARGET`` should match the triple
86+
used within the sysroot (i.e. ``$SYSROOT/usr/lib/$TARGET`` should exist).
11687

117-
#. The ARM libraries won't be installed in your system.
118-
But the CMake prepare step, which checks for
119-
dependencies, will check the *host* libraries, not the *target*
120-
ones. Below there's a list of some dependencies, but your project could
121-
have more, or this document could be outdated. You'll see the errors
122-
while linking as an indication of that.
88+
Then execute the following snippet to create a toolchain file:
12389

124-
Debian based distros have a way to add ``multiarch``, which adds
125-
a new architecture and allows you to install packages for those
126-
systems. See https://wiki.debian.org/Multiarch/HOWTO for more info.
90+
.. code-block:: bash
12791
128-
But not all distros will have that, and possibly not an easy way to
129-
install them in any anyway, so you'll have to build/download
130-
them separately.
92+
cat - <<EOF > $TARGET-clang.cmake
93+
set(CMAKE_SYSTEM_NAME Linux)
94+
set(CMAKE_SYSROOT "$SYSROOT")
95+
set(CMAKE_C_COMPILER_TARGET $TARGET)
96+
set(CMAKE_CXX_COMPILER_TARGET $TARGET)
97+
set(CMAKE_C_FLAGS_INIT "$CFLAGS")
98+
set(CMAKE_CXX_FLAGS_INIT "$CFLAGS")
99+
set(CMAKE_LINKER_TYPE LLD)
100+
set(CMAKE_C_COMPILER clang)
101+
set(CMAKE_CXX_COMPILER clang++)
102+
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
103+
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
104+
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
105+
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
106+
EOF
107+
108+
109+
Then configure and build by invoking ``cmake``:
131110
132-
A quick way of getting the libraries is to download them from
133-
a distribution repository, like Debian (http://packages.debian.org/jessie/),
134-
and download the missing libraries. Note that the ``libXXX``
135-
will have the shared objects (``.so``) and the ``libXXX-dev`` will
136-
give you the headers and the static (``.a``) library. Just in
137-
case, download both.
111+
.. code-block:: bash
138112
139-
The ones you need for ARM are: ``libtinfo``, ``zlib1g``,
140-
``libxml2`` and ``liblzma``. In the Debian repository you'll
141-
find downloads for all architectures.
113+
cmake -G Ninja \
114+
-DCMAKE_BUILD_TYPE=Release \
115+
-DLLVM_ENABLE_PROJECTS="lld;clang" \
116+
-DCMAKE_TOOLCHAIN_FILE=$(pwd)/$TARGET-clang.cmake \
117+
-DLLVM_HOST_TRIPLE=$TARGET \
118+
-DCMAKE_INSTALL_PREFIX=$HOME/clang-$TARGET \
119+
-S llvm \
120+
-B build/$TARGET
121+
cmake --build build/$TARGET
122+
123+
These options from the toolchain file and ``cmake`` invocation above are
124+
important:
125+
126+
* ``CMAKE_SYSTEM_NAME``: Perhaps surprisingly, explicitly setting this
127+
variable `causes CMake to set
128+
CMAKE_CROSSCOMPIILING <https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING.html#variable:CMAKE_CROSSCOMPILING>`_.
129+
* ``CMAKE_{C,CXX}_COMPILER_TARGET``: This will be used to set the
130+
``--target`` argument to ``clang``. The triple should match the triple used
131+
within the sysroot (i.e. ``$SYSROOT/usr/lib/$TARGET`` should exist).
132+
* ``CMAKE_FIND_ROOT_PATH_MODE_*``: These `control the search behaviour for
133+
finding libraries, includes or binaries
134+
<https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html#finding-external-libraries-programs-and-other-files>`_.
135+
Setting these prevents files for the host being used in the build.
136+
* ``LLVM_HOST_TRIPLE``: Specifies the target triple of the system the built
137+
LLVM will run on, which also implicitly sets other defaults such as
138+
``LLVM_DEFAULT_TARGET_TRIPLE``. For example, if you are using an x86_64
139+
host to compile for RISC-V, this will be a RISC-V triple.
140+
* ``CMAKE_SYSROOT``: The path to the sysroot containing libraries and headers
141+
for the target.
142+
* ``CMAKE_INSTALL_PREFIX``: Setting this avoids installing binaries compiled
143+
for the target system into system directories for the host system. It is
144+
not required unless you are going to use the ``install`` target.
145+
146+
See `LLVM's build documentation
147+
<https://llvm.org/docs/CMake.html#frequently-used-cmake-variables>`_ for more
148+
guidance on CMake variables (e.g. ``LLVM_TARGETS_TO_BUILD`` may be useful if
149+
your cross-compiled binaries only need to support compiling for one target).
150+
151+
Working around a ninja dependency issue
152+
---------------------------------------
153+
154+
If you followed the instructions above to create a sysroot, you may run into a
155+
`longstanding problem related to path canonicalization in ninja
156+
<https://github.com/ninja-build/ninja/issues/1330>_`. GCC canonicalizes system
157+
headers in dependency files, so when ninja reads them it does not need to do
158+
so. Clang does not do this, and unfortunately ninja does not implement the
159+
canonicalization logic at all, meaning for some system headers with symlinks
160+
in the paths, it can incorrectly compute a non-existing path and consider it
161+
as always modified.
162+
163+
If you are suffering from this issue, you will find any attempt at an
164+
incremental build (including the suggested command to build the ``install``
165+
target in the next section) results in recompiling everything. ``ninja -C
166+
build/$TARGET -t deps`` shows files in ``$SYSROOT/include/*`` that
167+
do not exist (as the ``$SYSROOT/include`` folder does not exist) and you can
168+
further confirm these files are causing ``ninja`` to determine a rebuild is
169+
necessary with ``ninja -C build/$TARGET -d deps``.
170+
171+
A workaround is to create a symlink so that the incorrect
172+
``$SYSROOT/include/*`` dependencies resolve to files within
173+
``$SYSROOT/usr/include/*``. This works in practice for the simple
174+
cross-compilation use case described here, but is not a general solution.
142175
143-
After you download and unpack all ``.deb`` packages, copy all
144-
``.so`` and ``.a`` to a directory, make the appropriate
145-
symbolic links (if necessary), and add the relevant ``-L``
146-
and ``-I`` paths to ``-DCMAKE_CXX_FLAGS`` above.
176+
.. code-block:: bash
147177
178+
sudo ln -s usr/include $SYSROOT/include
148179
149-
Running CMake and Building
150-
--------------------------
180+
Testing the just-built compiler
181+
-------------------------------
151182
152-
Finally, if you're using your platform compiler, run:
183+
Confirm the ``clang`` binary was built for the expected target architecture:
153184
154185
.. code-block:: bash
155186
156-
$ cmake -G Ninja <source-dir> -DCMAKE_BUILD_TYPE=<type> <options above>
187+
$ file -L ./build/aarch64-linux-gnu/bin/clang
188+
./build/aarch64-linux-gnu/bin/clang: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=516b8b366a790fcd3563bee4aec0cdfcb90bb1c7, not stripped
157189
158-
If you're using Clang as the cross-compiler, run:
190+
If you have ``qemu-user`` installed you can test the produced target binary
191+
either by invoking ``qemu-{target}-static`` directly:
159192
160193
.. code-block:: bash
161194
162-
$ CC='clang' CXX='clang++' cmake -G Ninja <source-dir> -DCMAKE_BUILD_TYPE=<type> <options above>
163-
164-
If you have ``clang``/``clang++`` on the path, it should just work, and special
165-
Ninja files will be created in the build directory. I strongly suggest
166-
you to run ``cmake`` on a separate build directory, *not* inside the
167-
source tree.
195+
$ qemu-aarch64-static -L $SYSROOT ./build/aarch64-linux-gnu/bin/clang --version
196+
clang version 21.0.0git (https://github.com/llvm/llvm-project cedfdc6e889c5c614a953ed1f44bcb45a405f8da)
197+
Target: aarch64-unknown-linux-gnu
198+
Thread model: posix
199+
InstalledDir: /home/asb/llvm-project/build/aarch64-linux-gnu/bin
168200
169-
To build, simply type:
201+
Or, if binfmt_misc is configured (as was necessary for debootstrap):
170202
171203
.. code-block:: bash
172204
173-
$ ninja
205+
$ export QEMU_LD_PREFIX=$SYSROOT; ./build/aarch64-linux-gnu/bin/clang --version
206+
clang version 21.0.0git (https://github.com/llvm/llvm-project cedfdc6e889c5c614a953ed1f44bcb45a405f8da)
207+
Target: aarch64-unknown-linux-gnu
208+
Thread model: posix
209+
InstalledDir: /home/asb/llvm-project/build/aarch64-linux-gnu/bin
174210
175-
It should automatically find out how many cores you have, what are
176-
the rules that needs building and will build the whole thing.
177-
178-
You can't run ``ninja check-all`` on this tree because the created
179-
binaries are targeted to ARM, not x86_64.
180-
181-
Installing and Using
211+
Installing and using
182212
--------------------
183213
184-
After the LLVM/Clang has built successfully, you should install it
185-
via:
186-
187-
.. code-block:: bash
188-
189-
$ ninja install
214+
.. note::
215+
Use of the ``install`` target requires that you have set
216+
``CMAKE_INSTALL_PREFIX`` otherwise it will attempt to install in
217+
directories under `/` on your host.
190218
191-
which will create a sysroot on the install-dir. You can then tar
192-
that directory into a binary with the full triple name (for easy
193-
identification), like:
219+
If you want to transfer a copy of the built compiler to another machine, you
220+
can first install it to a location on the host via:
194221
195222
.. code-block:: bash
196223
197-
$ ln -sf <install-dir> arm-linux-gnueabihf-clang
198-
$ tar zchf arm-linux-gnueabihf-clang.tar.gz arm-linux-gnueabihf-clang
224+
cmake --build build/$TARGET --target=install
199225
200-
If you copy that tarball to your target board, you'll be able to use
201-
it for running the test-suite, for example. Follow the guidelines at
202-
https://llvm.org/docs/lnt/quickstart.html, unpack the tarball in the
203-
test directory, and use options:
226+
This will install the LLVM/Clang headers, binaries, libraries, and other files
227+
to paths within ``CMAKE_INSTALL_PREFIX``. Then tar that directory for transfer
228+
to a device that runs the target architecture natively:
204229
205230
.. code-block:: bash
206231
207-
$ ./sandbox/bin/python sandbox/bin/lnt runtest nt \
208-
--sandbox sandbox \
209-
--test-suite `pwd`/test-suite \
210-
--cc `pwd`/arm-linux-gnueabihf-clang/bin/clang \
211-
--cxx `pwd`/arm-linux-gnueabihf-clang/bin/clang++
232+
tar -czvf clang-$TARGET.tar.gz -C $HOME clang-$TARGET
212233
213-
Remember to add the ``-jN`` options to ``lnt`` to the number of CPUs
214-
on your board. Also, the path to your clang has to be absolute, so
215-
you'll need the `pwd` trick above.
234+
The generated toolchain is portable, but requires compatible versions of any
235+
shared libraries it links against. This means using a sysroot that is as
236+
similar to your target operating system as possible is desirable. Other `CMake
237+
variables <https://llvm.org/docs/CMake.html#frequently-used-cmake-variables>`_
238+
may be helpful, for instance ``LLVM_STATIC_LINK_CXX_STDLIB``.

0 commit comments

Comments
 (0)