Skip to content

[Tools][Docker] Update Dockerfiles and Docker guide #123841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 15 additions & 21 deletions llvm/docs/Docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ You can find a number of sources to build docker images with LLVM components in
images for their own use, or as a starting point for someone who wants to write
their own Dockerfiles.

We currently provide Dockerfiles with ``debian10`` and ``nvidia-cuda`` base images.
We currently provide Dockerfiles with ``debian12`` and ``nvidia-cuda`` base images.
We also provide an ``example`` image, which contains placeholders that one would need
to fill out in order to produce Dockerfiles for a new docker image.

Expand Down Expand Up @@ -72,13 +72,13 @@ checkout from git and provide a list of CMake arguments to use during when
building LLVM inside docker container.

Here's a very simple example of getting a docker image with clang binary,
compiled by the system compiler in the debian10 image:
compiled by the system compiler in the debian12 image:

.. code-block:: bash

./llvm/utils/docker/build_docker_image.sh \
--source debian10 \
--docker-repository clang-debian10 --docker-tag "staging" \
--source debian12 \
--docker-repository clang-debian12 --docker-tag "staging" \
-p clang -i install-clang -i install-clang-resource-headers \
-- \
-DCMAKE_BUILD_TYPE=Release
Expand All @@ -93,51 +93,45 @@ this command will do that:
# LLVM_TARGETS_TO_BUILD=Native is to reduce stage1 compile time.
# Options, starting with BOOTSTRAP_* are passed to stage2 cmake invocation.
./build_docker_image.sh \
--source debian10 \
--docker-repository clang-debian10 --docker-tag "staging" \
--source debian12 \
--docker-repository clang-debian12 --docker-tag "staging" \
-p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \
-- \
-DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \
-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-resource-headers"

This will produce a new image ``clang-debian10:staging`` from the latest
This will produce a new image ``clang-debian12:staging`` from the latest
upstream revision.
After the image is built you can run bash inside a container based on your image
like this:

.. code-block:: bash

docker run -ti clang-debian10:staging bash
docker run -ti clang-debian12:staging bash

Now you can run bash commands as you normally would:

.. code-block:: bash

root@80f351b51825:/# clang -v
clang version 5.0.0 (trunk 305064)
clang version 19.1.7 (trunk 524462)
Target: x86_64-unknown-linux-gnu
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Candidate multilib: .;@m64
Selected multilib: .;@m64


Which image should I choose?
============================
We currently provide two images: Debian10-based and nvidia-cuda-based. They
We currently provide two images: Debian12-based and nvidia-cuda-based. They
differ in the base image that they use, i.e. they have a different set of
preinstalled binaries. Debian8 is very minimal, nvidia-cuda is larger, but has
preinstalled CUDA libraries and allows to access a GPU, installed on your
machine.

If you need a minimal linux distribution with only clang and libstdc++ included,
you should try Debian10-based image.
you should try Debian12-based image.

If you want to use CUDA libraries and have access to a GPU on your machine,
you should choose nvidia-cuda-based image and use `nvidia-docker
Expand All @@ -150,20 +144,20 @@ If you have a different use-case, you could create your own image based on
``example/`` folder.

Any docker image can be built and run using only the docker binary, i.e. you can
run debian10 build on Fedora or any other Linux distribution. You don't need to
run debian12 build on Fedora or any other Linux distribution. You don't need to
install CMake, compilers or any other clang dependencies. It is all handled
during the build process inside Docker's isolated environment.

Stable build
============
If you want a somewhat recent and somewhat stable build, use the
``branches/google/stable`` branch, i.e. the following command will produce a
Debian10-based image using the latest ``google/stable`` sources for you:
Debian12-based image using the latest ``google/stable`` sources for you:

.. code-block:: bash

./llvm/utils/docker/build_docker_image.sh \
-s debian10 --d clang-debian10 -t "staging" \
-s debian12 --d clang-debian12 -t "staging" \
--branch branches/google/stable \
-p clang -i install-clang -i install-clang-resource-headers \
-- \
Expand Down
16 changes: 8 additions & 8 deletions llvm/utils/docker/build_docker_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Available options:
General:
-h|--help show this help message
Docker-specific:
-s|--source image source dir (i.e. debian10, nvidia-cuda, etc)
-s|--source image source dir (i.e. debian12, nvidia-cuda, etc)
-d|--docker-repository docker repository for the image
-t|--docker-tag docker tag for the image
Checkout arguments:
Expand All @@ -38,7 +38,7 @@ Available options:
-p|--llvm-project Add the project to a list LLVM_ENABLE_PROJECTS, passed to
CMake.
Can be specified multiple times.
-c|--checksums name of a file, containing checksums of llvm checkout.
--checksums name of a file, containing checksums of llvm checkout.
Script will fail if checksums of the checkout do not
match.
Build-specific:
Expand All @@ -51,18 +51,18 @@ Required options: --source and --docker-repository, at least one
All options after '--' are passed to CMake invocation.

For example, running:
$ build_docker_image.sh -s debian10 -d mydocker/debian10-clang -t latest \
$ build_docker_image.sh -s debian12 -d mydocker/debian12-clang -t latest \
-p clang -i install-clang -i install-clang-resource-headers
will produce two docker images:
mydocker/debian10-clang-build:latest - an intermediate image used to compile
mydocker/debian12-clang-build:latest - an intermediate image used to compile
clang.
mydocker/clang-debian10:latest - a small image with preinstalled clang.
mydocker/clang-debian12:latest - a small image with preinstalled clang.
Please note that this example produces a not very useful installation, since it
doesn't override CMake defaults, which produces a Debug and non-boostrapped
version of clang.

To get a 2-stage clang build, you could use this command:
$ ./build_docker_image.sh -s debian10 -d mydocker/clang-debian10 -t "latest" \
$ ./build_docker_image.sh -s debian12 -d mydocker/clang-debian12 -t "latest" \
-p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \
-- \
-DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \
Expand Down Expand Up @@ -110,7 +110,7 @@ while [[ $# -gt 0 ]]; do
CMAKE_ENABLED_PROJECTS="$CMAKE_ENABLED_PROJECTS;$PROJ"
shift 2
;;
-c|--checksums)
--checksums)
shift
CHECKSUMS_FILE="$1"
shift
Expand Down Expand Up @@ -160,7 +160,7 @@ if [ $SEEN_INSTALL_TARGET -eq 0 ]; then
exit 1
fi

SOURCE_DIR=$(dirname $0)
SOURCE_DIR=$(dirname "$0")
if [ ! -d "$SOURCE_DIR/$IMAGE_SOURCE" ]; then
echo "No sources for '$IMAGE_SOURCE' were found in $SOURCE_DIR"
exit 1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
#===- llvm/utils/docker/debian10/build/Dockerfile -------------------------===//
#===- llvm/utils/docker/debian12/build/Dockerfile -------------------------===//
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
#===----------------------------------------------------------------------===//
# Stage 1. Check out LLVM source code and run the build.
FROM launcher.gcr.io/google/debian10:latest as builder
LABEL maintainer "LLVM Developers"
FROM launcher.gcr.io/google/debian12:latest AS builder
LABEL maintainer="LLVM Developers"
# Install build dependencies of llvm.
# First, Update the apt's source list and include the sources of the packages.
RUN grep deb /etc/apt/sources.list | \
Expand All @@ -19,8 +19,8 @@ RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
# Install a newer ninja release. It seems the older version in the debian repos
# randomly crashes when compiling llvm.
RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip" && \
echo "6f98805688d19672bd699fbbfa2c2cf0fc054ac3df1f0e6a47664d963d530255 ninja-linux.zip" \
| sha256sum -c && \
unzip ninja-linux.zip -d /usr/local/bin && \
rm ninja-linux.zip
Expand All @@ -37,11 +37,11 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg


# Stage 2. Produce a minimal release image with build results.
FROM launcher.gcr.io/google/debian10:latest
LABEL maintainer "LLVM Developers"
FROM launcher.gcr.io/google/debian12:latest
LABEL maintainer="LLVM Developers"
# Install packages for minimal useful image.
RUN apt-get update && \
apt-get install -y --no-install-recommends libstdc++-7-dev binutils && \
apt-get install -y --no-install-recommends binutils && \
rm -rf /var/lib/apt/lists/*
# Copy build results of stage 1 to /usr/local.
COPY --from=builder /tmp/clang-install/ /usr/local/
6 changes: 3 additions & 3 deletions llvm/utils/docker/example/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

# Stage 1. Check out LLVM source code and run the build.
# FIXME: Replace 'ubuntu' with your base image
FROM ubuntu as builder
FROM ubuntu AS builder
# FIXME: Change maintainer name
LABEL maintainer "Maintainer <maintainer@email>"
LABEL maintainer="Maintainer <maintainer@email>"
# FIXME: Install llvm/clang build dependencies here. Including compiler to
# build stage1, cmake, subversion, ninja, etc.

Expand All @@ -31,7 +31,7 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg
# FIXME: Replace 'ubuntu' with your base image.
FROM ubuntu
# FIXME: Change maintainer name.
LABEL maintainer "Maintainer <maintainer@email>"
LABEL maintainer="Maintainer <maintainer@email>"
# FIXME: Install all packages you want to have in your release container.
# A minimal useful installation should include at least libstdc++ and binutils.

Expand Down
10 changes: 5 additions & 5 deletions llvm/utils/docker/nvidia-cuda/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@
#
#===----------------------------------------------------------------------===//
# Stage 1. Check out LLVM source code and run the build.
FROM nvidia/cuda:8.0-devel as builder
LABEL maintainer "LLVM Developers"
FROM nvidia/cuda:12.6.3-devel-ubuntu24.04 AS builder
LABEL maintainer="LLVM Developers"
# Install llvm build dependencies.
RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates cmake python \
apt-get install -y --no-install-recommends ca-certificates cmake 2to3 python-is-python3 \
subversion ninja-build git && \
rm -rf /var/lib/apt/lists/*

Expand All @@ -26,8 +26,8 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg


# Stage 2. Produce a minimal release image with build results.
FROM nvidia/cuda:8.0-devel
LABEL maintainer "LLVM Developers"
FROM nvidia/cuda:12.6.3-devel-ubuntu24.04
LABEL maintainer="LLVM Developers"
# Copy clang installation into this container.
COPY --from=builder /tmp/clang-install/ /usr/local/
# C++ standard library and binutils are already included in the base package.
8 changes: 4 additions & 4 deletions llvm/utils/docker/scripts/checkout.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Checkout git sources into /tmp/clang-build/src. Used inside a docker container.
Available options:
-h|--help show this help message
-b|--branch git branch to checkout, i.e. 'main',
'release/10.x'
'release/19.x'
(default: 'main')
-r|--revision git revision to checkout
-c|--cherrypick revision to cherry-pick. Can be specified multiple times.
Expand Down Expand Up @@ -83,7 +83,7 @@ function apply_cherrypicks() {
# This function is always called on a sorted list of cherrypicks.
for CHERRY_REV in $CHERRYPICKS; do
echo "Cherry-picking $CHERRY_REV into $CHECKOUT_DIR"
EMAIL="[email protected]" git cherry-pick $CHERRY_REV
EMAIL="[email protected]" git cherry-pick "$CHERRY_REV"
done

popd
Expand All @@ -97,7 +97,7 @@ mkdir -p "$CLANG_BUILD_DIR/src"
CHECKOUT_DIR="$CLANG_BUILD_DIR/src"

echo "Checking out https://github.com/llvm/llvm-project.git to $CHECKOUT_DIR"
git clone -b $LLVM_BRANCH --single-branch \
git clone -b "$LLVM_BRANCH" --single-branch \
"https://github.com/llvm/llvm-project.git" \
"$CHECKOUT_DIR"

Expand All @@ -114,7 +114,7 @@ CHECKSUMS_FILE="/tmp/checksums/checksums.txt"

if [ -f "$CHECKSUMS_FILE" ]; then
echo "Validating checksums for LLVM checkout..."
python "$(dirname $0)/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \
python "$(dirname "$0")/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \
--partial --multi_dir "$CLANG_BUILD_DIR/src"
else
echo "Skipping checksumming checks..."
Expand Down
Loading