Skip to content

Commit f4e3fb5

Browse files
EricWFEricWF
authored
Refactor dockerfile to support Buildkite AND Github Actions (#71954)
This change adds the image used by the self-hosted Github Actions builders. In an attempt to make the transition simple, all of the different images share as much of the same state as possible, including packages, users, etc... This results in bigger images, but that shouldn't be a problem. That said, the refactorings caused the buildkite image to shrink by 100 MB. This change also renames all of the packages for consistency. Bots will have to be changed to use the new package names eventually. Again, docker-compose was used as the source of truth for defining argument. I have already pushed example images to ghcr.io/libcxx/<name>:testing
1 parent 2fbd088 commit f4e3fb5

File tree

4 files changed

+287
-182
lines changed

4 files changed

+287
-182
lines changed

‎libcxx/utils/ci/Dockerfile

Lines changed: 255 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,29 @@
55
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
66
#
77
#===----------------------------------------------------------------------===##
8-
98
#
10-
# This Dockerfile describes the base image used to run the various libc++
11-
# build bots. By default, the image runs the Buildkite Agent, however one
12-
# can also just start the image with a shell to debug CI failures.
9+
# This file defines the buildkite and github actions builder images.
10+
# You can build & push both images using:
11+
#
12+
# docker compose build
13+
# docker compose push
14+
#
15+
# Or you can select a single image to build & push using:
16+
#
17+
# docker compose build buildkite-builder
18+
# docker compose push buildkite-builder
19+
#
20+
# The final images can be found at
21+
#
22+
# ghcr.io/libcxx/buildkite-builder
23+
# ghcr.io/libcxx/actions-builder
24+
# ghcr.io/libcxx/android-buildkite-builder
25+
#
26+
# Members of the github.com/libcxx/ organizations have permissions required to push new images.
27+
#
28+
# ===----------------------------------------------------------------------===##
29+
# Running the buildkite image
30+
# ===----------------------------------------------------------------------===##
1331
#
1432
# To start a Buildkite Agent, run it as:
1533
# $ docker run --env-file <secrets> -it $(docker build -q libcxx/utils/ci)
@@ -21,42 +39,82 @@
2139
#
2240
# If you're only looking to run the Docker image locally for debugging a
2341
# build bot, see the `run-buildbot-container` script located in this directory.
24-
#
25-
# A pre-built version of this image is maintained on Github under the libc++ organization, as ghcr.io/libcxx/libcxx-builder.
26-
# To update the image, rebuild it and push it to github (all members of the libc++ organization should be able to do this).
27-
#
28-
# $ docker compose build
29-
# $ docker compose push
30-
#
3142

32-
FROM ubuntu:jammy
43+
44+
# HACK: We set the base image in the docker-compose file depending on the final target (buildkite vs github actions).
45+
# This means we have a much slower container build, but we can use the same Dockerfile for both targets.
46+
ARG BASE_IMAGE
47+
FROM $BASE_IMAGE AS builder-base
3348

3449
# Make sure apt-get doesn't try to prompt for stuff like our time zone, etc.
3550
ENV DEBIAN_FRONTEND=noninteractive
3651

37-
RUN apt-get update && apt-get install -y bash curl
52+
# populated in the docker-compose file
53+
ARG GCC_LATEST_VERSION
54+
ENV GCC_LATEST_VERSION=${GCC_LATEST_VERSION}
55+
56+
# populated in the docker-compose file
57+
ARG LLVM_HEAD_VERSION
58+
ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
59+
60+
# HACK: The github actions runner image already has sudo and requires its use. The buildkite base image does not.
61+
# Reconcile this.
62+
RUN <<EOF
63+
apt-get update || true
64+
apt-get install -y sudo || true
65+
echo "ALL ALL = (ALL) NOPASSWD: ALL" | tee /etc/sudoers || true
66+
EOF
67+
68+
RUN sudo apt-get update \
69+
&& sudo apt-get install -y \
70+
python3 \
71+
python3-distutils \
72+
python3-psutil \
73+
git \
74+
gdb \
75+
ccache \
76+
gpg \
77+
wget \
78+
bash \
79+
curl \
80+
python3 \
81+
python3-dev \
82+
libpython3-dev \
83+
uuid-dev \
84+
libncurses5-dev \
85+
swig3.0 \
86+
libxml2-dev \
87+
libedit-dev \
88+
language-pack-en \
89+
language-pack-fr \
90+
language-pack-ja \
91+
language-pack-ru \
92+
language-pack-zh-hans \
93+
lsb-release \
94+
wget \
95+
unzip \
96+
software-properties-common \
97+
&& sudo rm -rf /var/lib/apt/lists/*
98+
3899

39100
# Install various tools used by the build or the test suite
40101
#RUN apt-get update && apt-get install -y ninja-build python3 python3-distutils python3-psutil git gdb ccache
41102
# TODO add ninja-build once 1.11 is available in Ubuntu, also remove the manual installation.
42-
RUN apt-get update && apt-get install -y python3 python3-distutils python3-psutil git gdb ccache
43-
RUN apt-get update && apt-get install -y wget && \
44-
wget -qO /usr/local/bin/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip && \
45-
gunzip /usr/local/bin/ninja.gz && \
46-
chmod a+x /usr/local/bin/ninja
47-
48-
# Install dependencies required to run the LLDB data formatter tests
49-
RUN apt-get update && apt-get install -y python3 python3-dev libpython3-dev uuid-dev libncurses5-dev swig3.0 libxml2-dev libedit-dev
50-
51-
# Locales for gdb and localization tests
52-
RUN apt-get update && apt-get install -y language-pack-en language-pack-fr \
53-
language-pack-ja language-pack-ru \
54-
language-pack-zh-hans
55-
# These two are not enabled by default so generate them
56-
RUN printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" >> /etc/locale.gen
57-
RUN mkdir /usr/local/share/i1en/
58-
RUN printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" >> /usr/local/share/i1en/SUPPORTED
59-
RUN locale-gen
103+
RUN <<EOF
104+
wget -qO /tmp/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip
105+
gunzip /tmp/ninja.gz
106+
chmod a+x /tmp/ninja
107+
sudo mv /tmp/ninja /usr/local/bin/ninja
108+
EOF
109+
110+
111+
# These two locales are not enabled by default so generate them
112+
RUN <<EOF
113+
printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /etc/locale.gen
114+
sudo mkdir /usr/local/share/i1en/
115+
printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /usr/local/share/i1en/SUPPORTED
116+
sudo locale-gen
117+
EOF
60118

61119
# Install Clang <latest>, <latest-1> and ToT, which are the ones we support.
62120
# We also install <latest-2> because we need to support the "latest-1" of the
@@ -65,75 +123,179 @@ RUN locale-gen
65123
# LLVM 15, we still need to have Clang 12 in this Docker image because the LLVM
66124
# 14 release branch CI uses it. The tip-of-trunk CI will never use Clang 12,
67125
# though.
68-
ARG LLVM_HEAD_VERSION # populated in the docker-compose file
69-
ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
70-
RUN apt-get update && apt-get install -y lsb-release wget software-properties-common
71-
RUN wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
72-
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) # for CI transitions
73-
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) # previous release
74-
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) # latest release
75-
RUN bash /tmp/llvm.sh $LLVM_HEAD_VERSION # current ToT
76-
77-
# Install clang-scan-deps, which is required to build modules; always all supported versions.
78-
RUN apt-get update && apt-get install -y clang-tools-$(($LLVM_HEAD_VERSION - 3)) \
79-
clang-tools-$(($LLVM_HEAD_VERSION - 2)) \
80-
clang-tools-$(($LLVM_HEAD_VERSION - 1)) \
81-
clang-tools-$LLVM_HEAD_VERSION
82-
83-
# Install clang-format; always use the lastest stable branch.
84-
RUN apt-get update && apt-get install -y clang-format-$(($LLVM_HEAD_VERSION - 2)) clang-format-$(($LLVM_HEAD_VERSION - 1))
85-
86-
# Install clang-tidy
87-
# TODO(LLVM-17) revert D148831 to only install $(($LLVM_HEAD_VERSION - 1)) and $LLVM_HEAD_VERSION
88-
# The usage of the ToT version is needed due to module issues with Clang 16
89-
RUN apt-get update && apt-get install -y clang-tidy-$(($LLVM_HEAD_VERSION - 2)) clang-tidy-$(($LLVM_HEAD_VERSION - 1)) clang-tidy-$LLVM_HEAD_VERSION
90-
91-
# Install llvm-dev and libclang-dev to compile custom clang-tidy checks
92-
# TODO(LLVM-17) revert D148831 to only install $(($LLVM_HEAD_VERSION - 1)) and $LLVM_HEAD_VERSION
93-
# The usage of the ToT version is needed due to module issues with Clang 16
94-
RUN apt-get update && apt-get install -y llvm-$(($LLVM_HEAD_VERSION - 2))-dev llvm-$(($LLVM_HEAD_VERSION - 1))-dev llvm-$LLVM_HEAD_VERSION-dev \
95-
libclang-$(($LLVM_HEAD_VERSION - 2))-dev libclang-$(($LLVM_HEAD_VERSION - 1))-dev libclang-$LLVM_HEAD_VERSION-dev \
96-
libomp5-$LLVM_HEAD_VERSION
126+
RUN <<EOF
127+
sudo apt-get update
128+
wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
129+
chmod +x /tmp/llvm.sh
130+
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) all # for CI transitions
131+
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) all # previous release
132+
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) all # latest release
133+
sudo /tmp/llvm.sh $LLVM_HEAD_VERSION all # current ToT
134+
sudo apt-get install -y libomp5-$LLVM_HEAD_VERSION
135+
sudo rm -rf /var/lib/apt/lists/*
136+
EOF
97137

98138
# Install the most recent GCC, like clang install the previous version as a transition.
99-
RUN add-apt-repository ppa:ubuntu-toolchain-r/test
100-
ARG GCC_LATEST_VERSION # populated in the docker-compose file
101-
ENV GCC_LATEST_VERSION=${GCC_LATEST_VERSION}
102-
RUN apt-get update && apt install -y gcc-$((GCC_LATEST_VERSION - 1)) g++-$((GCC_LATEST_VERSION - 1))
103-
RUN apt-get update && apt install -y gcc-$GCC_LATEST_VERSION g++-$GCC_LATEST_VERSION
104-
105-
# Remove all, no longer needed, apt data
106-
RUN rm -rf /var/lib/apt/lists/*
107-
108-
# Install a recent CMake
109-
RUN wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
110-
RUN bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
111-
RUN rm /tmp/install-cmake.sh
112-
113-
# Install a newer CMake for modules
114-
# TODO Remove the duplicated installation when all runtimes can be build with CMake 3.28.
115-
RUN wget https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64.sh -O /tmp/install-cmake.sh
116-
RUN bash /tmp/install-cmake.sh --prefix=/opt --exclude-subdir --skip-license
117-
RUN rm /tmp/install-cmake.sh
118-
119-
RUN wget https://github.com/Kitware/CMake/releases/download/v3.28.0-rc4/cmake-3.28.0-rc4-linux-x86_64.sh -O /tmp/install-cmake.sh
120-
RUN mkdir /opt/cmake-3.28
121-
RUN bash /tmp/install-cmake.sh --prefix=/opt/cmake-3.28 --exclude-subdir --skip-license
122-
RUN rm /tmp/install-cmake.sh
123-
124-
# Change the user to a non-root user, since some of the libc++ tests
125-
# (e.g. filesystem) require running as non-root. Also setup passwordless sudo.
126-
RUN apt-get update && apt-get install -y sudo
127-
RUN echo "ALL ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers
128-
RUN useradd --create-home libcxx-builder
139+
RUN <<EOF
140+
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
141+
sudo apt-get update
142+
sudo apt-get install -y \
143+
gcc-$((GCC_LATEST_VERSION - 1)) \
144+
g++-$((GCC_LATEST_VERSION - 1)) \
145+
gcc-$GCC_LATEST_VERSION \
146+
g++-$GCC_LATEST_VERSION
147+
sudo rm -rf /var/lib/apt/lists/*
148+
EOF
149+
150+
RUN <<EOF
151+
# Install a recent CMake
152+
wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
153+
sudo bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
154+
rm /tmp/install-cmake.sh
155+
156+
# Install a newer CMake for modules
157+
# TODO Remove the duplicated installation when all runtimes can be build with CMake 3.28.
158+
wget https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64.sh -O /tmp/install-cmake.sh
159+
sudo bash /tmp/install-cmake.sh --prefix=/opt --exclude-subdir --skip-license
160+
rm /tmp/install-cmake.sh
161+
162+
wget https://github.com/Kitware/CMake/releases/download/v3.28.0-rc4/cmake-3.28.0-rc4-linux-x86_64.sh -O /tmp/install-cmake.sh
163+
sudo mkdir /opt/cmake-3.28
164+
sudo bash /tmp/install-cmake.sh --prefix=/opt/cmake-3.28 --exclude-subdir --skip-license
165+
rm /tmp/install-cmake.sh
166+
EOF
167+
168+
# ===----------------------------------------------------------------------===##
169+
# Android Buildkite Image
170+
# ===----------------------------------------------------------------------===##
171+
172+
FROM ubuntu:jammy AS android-builder-base
173+
174+
ARG ANDROID_CLANG_VERSION
175+
ARG ANDROID_CLANG_PREBUILTS_COMMIT
176+
ARG ANDROID_SYSROOT_BID
177+
178+
RUN apt-get update && apt-get install -y curl unzip git
179+
180+
# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
181+
RUN <<EOF
182+
mkdir -p /opt/android/sdk
183+
cd /opt/android/sdk
184+
curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip
185+
unzip platform-tools-latest-linux.zip
186+
rm platform-tools-latest-linux.zip
187+
EOF
188+
189+
# Install the current Android compiler. Specify the prebuilts commit to retrieve
190+
# this compiler version even after it's removed from HEAD.
191+
192+
ENV ANDROID_CLANG_VERSION=$ANDROID_CLANG_VERSION
193+
ENV ANDROID_CLANG_PREBUILTS_COMMIT=$ANDROID_CLANG_PREBUILTS_COMMIT
194+
RUN <<EOF
195+
git clone --filter=blob:none --sparse \
196+
https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
197+
/opt/android/clang
198+
git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT}
199+
git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION}
200+
rm -fr /opt/android/clang/.git
201+
ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current
202+
# The "git sparse-checkout" and "ln" commands succeed even if nothing was
203+
# checked out, so use this "ls" command to fix that.
204+
ls /opt/android/clang/clang-current/bin/clang
205+
EOF
206+
207+
# Install an Android sysroot. New AOSP sysroots are available at
208+
# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
209+
# NDK also makes its sysroot prebuilt available at
210+
# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
211+
212+
ENV ANDROID_SYSROOT_BID=$ANDROID_SYSROOT_BID
213+
RUN <<EOF
214+
cd /opt/android
215+
curl -L -o ndk_platform.tar.bz2 \
216+
https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url
217+
tar xf ndk_platform.tar.bz2
218+
rm ndk_platform.tar.bz2
219+
EOF
220+
221+
# Install Docker
222+
RUN <<EOF
223+
curl -fsSL https://get.docker.com -o /tmp/get-docker.sh
224+
sh /tmp/get-docker.sh
225+
rm /tmp/get-docker.sh
226+
227+
# Install Docker. Mark the binary setuid so it can be run without prefixing it
228+
# with sudo. Adding the container user to the docker group doesn't work because
229+
# /var/run/docker.sock is owned by the host's docker GID, not the container's
230+
# docker GID.
231+
chmod u+s /usr/bin/docker
232+
EOF
233+
234+
# ===----------------------------------------------------------------------===##
235+
# Buildkite Builder Image
236+
# ===----------------------------------------------------------------------===##
237+
#
238+
# IMAGE: ghcr.io/libcxx/buildkite-builder.
239+
#
240+
FROM builder-base AS buildkite-builder
241+
242+
# Create the libcxx-builder user, regardless of if we use it or not
243+
RUN sudo useradd --create-home libcxx-builder
244+
129245
USER libcxx-builder
130246
WORKDIR /home/libcxx-builder
131247

132248
# Install the Buildkite agent and dependencies. This must be done as non-root
133249
# for the Buildkite agent to be installed in a path where we can find it.
134-
RUN bash -c "$(curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh)"
250+
RUN <<EOF
251+
cd /home/libcxx-builder
252+
curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh -o /tmp/install-agent.sh
253+
bash /tmp/install-agent.sh
254+
rm /tmp/install-agent.sh
255+
echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" \
256+
>> /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
257+
EOF
258+
259+
USER libcxx-builder
260+
WORKDIR /home/libcxx-builder
261+
135262
ENV PATH="${PATH}:/home/libcxx-builder/.buildkite-agent/bin"
136-
RUN echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" >> "/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg"
137263

138-
# By default, start the Buildkite agent (this requires a token).
139264
CMD ["buildkite-agent", "start"]
265+
266+
# ===----------------------------------------------------------------------===##
267+
# Android Buildkite Builder Image
268+
# ===----------------------------------------------------------------------===##
269+
#
270+
# IMAGE: ghcr.io/libcxx/android-buildkite-builder.
271+
#
272+
FROM buildkite-builder AS android-buildkite-builder
273+
274+
COPY --from=android-builder-base /opt/android /opt/android
275+
COPY ./vendor/android/container-setup.sh /opt/android/container-setup.sh
276+
277+
ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
278+
279+
USER libcxx-builder
280+
WORKDIR /home/libcxx-builder
281+
282+
# Reset the configuration, we pass the configuration via the environment.
283+
RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
284+
/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
285+
286+
# Modify the Buildkite agent cmdline to do Android setup stuff first.
287+
CMD /opt/android/container-setup.sh && buildkite-agent start
288+
289+
# ===----------------------------------------------------------------------===##
290+
# Github Actions Builder Image
291+
# ===----------------------------------------------------------------------===##
292+
#
293+
# IMAGE: ghcr.io/libcxx/actions-builder.
294+
#
295+
FROM builder-base AS actions-builder
296+
297+
WORKDIR /home/runner
298+
USER runner
299+
300+
301+

0 commit comments

Comments
 (0)