Skip to content

Commit 38bd381

Browse files
committed
Auto merge of #47828 - alexcrichton:llvm-6, r=<try>
rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3
2 parents 771873c + d9031ae commit 38bd381

File tree

28 files changed

+279
-133
lines changed

28 files changed

+279
-133
lines changed

.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,6 @@
5151
[submodule "src/doc/rust-by-example"]
5252
path = src/doc/rust-by-example
5353
url = https://github.com/rust-lang/rust-by-example
54+
[submodule "src/llvm-emscripten"]
55+
path = src/llvm-emscripten
56+
url = https://github.com/rust-lang/llvm

.travis.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ matrix:
8181
# OSX 10.7 and `xcode7` is the latest Xcode able to compile LLVM for 10.7.
8282
- env: >
8383
RUST_CHECK_TARGET=dist
84-
RUST_CONFIGURE_ARGS="--build=i686-apple-darwin --enable-extended --enable-profiler"
84+
RUST_CONFIGURE_ARGS="--build=i686-apple-darwin --enable-extended --enable-profiler --enable-emscripten"
8585
SRC=.
8686
DEPLOY=1
8787
RUSTC_RETRY_LINKER_ON_SEGFAULT=1
@@ -95,7 +95,7 @@ matrix:
9595
9696
- env: >
9797
RUST_CHECK_TARGET=dist
98-
RUST_CONFIGURE_ARGS="--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler"
98+
RUST_CONFIGURE_ARGS="--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler --enable-emscripten"
9999
SRC=.
100100
DEPLOY=1
101101
RUSTC_RETRY_LINKER_ON_SEGFAULT=1

appveyor.yml

+4-2
Original file line numberDiff line numberDiff line change
@@ -63,25 +63,27 @@ environment:
6363
--build=x86_64-pc-windows-msvc
6464
--enable-extended
6565
--enable-profiler
66+
--enable-emscripten
6667
SCRIPT: python x.py dist
6768
DEPLOY: 1
6869
- RUST_CONFIGURE_ARGS: >
6970
--build=i686-pc-windows-msvc
7071
--target=i586-pc-windows-msvc
7172
--enable-extended
7273
--enable-profiler
74+
--enable-emscripten
7375
SCRIPT: python x.py dist
7476
DEPLOY: 1
7577
- MSYS_BITS: 32
76-
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu --enable-extended
78+
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu --enable-extended --enable-emscripten
7779
SCRIPT: python x.py dist
7880
MINGW_URL: https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror
7981
MINGW_ARCHIVE: i686-6.3.0-release-posix-dwarf-rt_v5-rev2.7z
8082
MINGW_DIR: mingw32
8183
DEPLOY: 1
8284
- MSYS_BITS: 64
8385
SCRIPT: python x.py dist
84-
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu --enable-extended
86+
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu --enable-extended --enable-emscripten
8587
MINGW_URL: https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror
8688
MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z
8789
MINGW_DIR: mingw64

config.toml.example

+7
Original file line numberDiff line numberDiff line change
@@ -305,6 +305,13 @@
305305
# result (broken, compiling, testing) into this JSON file.
306306
#save-toolstates = "/path/to/toolstates.json"
307307

308+
# This is an array of the codegen backends that will be compiled for the rustc
309+
# that's being compiled. The default is to only build the LLVM codegen backend,
310+
# but you can also optionally enable the "emscripten" backend for asm.js or
311+
# make this an empty array (but that probably won't get too far in the
312+
# bootstrap)
313+
#codegen-backends = ["llvm"]
314+
308315
# =============================================================================
309316
# Options for specific targets
310317
#

src/bootstrap/bootstrap.py

+16-7
Original file line numberDiff line numberDiff line change
@@ -640,14 +640,23 @@ def update_submodules(self):
640640
os.path.join(self.rust_root, ".gitmodules"),
641641
"--get-regexp", "path"]
642642
).decode(default_encoding).splitlines()]
643-
submodules = [module for module in submodules
644-
if not ((module.endswith("llvm") and
645-
self.get_toml('llvm-config')) or
646-
(module.endswith("jemalloc") and
647-
(self.get_toml('use-jemalloc') == "false" or
648-
self.get_toml('jemalloc'))))]
643+
filtered_submodules = []
644+
for module in submodules:
645+
if module.endswith("llvm"):
646+
if self.get_toml('llvm-config'):
647+
continue
648+
if module.endswith("llvm-emscripten"):
649+
backends = self.get_toml('codegen-backends')
650+
if backends is None or not 'emscripten' in backends:
651+
continue
652+
if module.endswith("jemalloc"):
653+
if self.get_toml('use-jemalloc') == 'false':
654+
continue
655+
if self.get_toml('jemalloc'):
656+
continue
657+
filtered_submodules.append(module)
649658
run(["git", "submodule", "update",
650-
"--init", "--recursive"] + submodules,
659+
"--init", "--recursive"] + filtered_submodules,
651660
cwd=self.rust_root, verbose=self.verbose)
652661
run(["git", "submodule", "-q", "foreach", "git",
653662
"reset", "-q", "--hard"],

src/bootstrap/compile.rs

+119-75
Original file line numberDiff line numberDiff line change
@@ -581,24 +581,30 @@ impl Step for RustcLink {
581581
}
582582

583583
#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
584-
pub struct RustcTrans {
584+
pub struct CodegenBackend {
585585
pub compiler: Compiler,
586586
pub target: Interned<String>,
587+
pub backend: Interned<String>,
587588
}
588589

589-
impl Step for RustcTrans {
590+
impl Step for CodegenBackend {
590591
type Output = ();
591592
const ONLY_HOSTS: bool = true;
592593
const DEFAULT: bool = true;
593594

594595
fn should_run(run: ShouldRun) -> ShouldRun {
595-
run.path("src/librustc_trans").krate("rustc_trans")
596+
run.path("src/librustc_trans")
596597
}
597598

598599
fn make_run(run: RunConfig) {
599-
run.builder.ensure(RustcTrans {
600+
let backend = run.builder.config.rust_codegen_backends.get(0);
601+
let backend = backend.cloned().unwrap_or_else(|| {
602+
INTERNER.intern_str("llvm")
603+
});
604+
run.builder.ensure(CodegenBackend {
600605
compiler: run.builder.compiler(run.builder.top_stage, run.host),
601606
target: run.target,
607+
backend
602608
});
603609
}
604610

@@ -609,58 +615,92 @@ impl Step for RustcTrans {
609615

610616
builder.ensure(Rustc { compiler, target });
611617

612-
// Build LLVM for our target. This will implicitly build the host LLVM
613-
// if necessary.
614-
builder.ensure(native::Llvm { target });
615-
616618
if build.force_use_stage1(compiler, target) {
617-
builder.ensure(RustcTrans {
619+
builder.ensure(CodegenBackend {
618620
compiler: builder.compiler(1, build.build),
619621
target,
622+
backend: self.backend,
620623
});
621624
return;
622625
}
623626

624-
let _folder = build.fold_output(|| format!("stage{}-rustc_trans", compiler.stage));
625-
println!("Building stage{} trans artifacts ({} -> {})",
626-
compiler.stage, &compiler.host, target);
627-
628627
let mut cargo = builder.cargo(compiler, Mode::Librustc, target, "build");
628+
let mut features = build.rustc_features().to_string();
629629
cargo.arg("--manifest-path")
630-
.arg(build.src.join("src/librustc_trans/Cargo.toml"))
631-
.arg("--features").arg(build.rustc_features());
630+
.arg(build.src.join("src/librustc_trans/Cargo.toml"));
632631
rustc_cargo_env(build, &mut cargo);
633632

634-
// Pass down configuration from the LLVM build into the build of
635-
// librustc_llvm and librustc_trans.
636-
if build.is_rust_llvm(target) {
637-
cargo.env("LLVM_RUSTLLVM", "1");
638-
}
639-
cargo.env("LLVM_CONFIG", build.llvm_config(target));
640-
let target_config = build.config.target_config.get(&target);
641-
if let Some(s) = target_config.and_then(|c| c.llvm_config.as_ref()) {
642-
cargo.env("CFG_LLVM_ROOT", s);
643-
}
644-
// Building with a static libstdc++ is only supported on linux right now,
645-
// not for MSVC or macOS
646-
if build.config.llvm_static_stdcpp &&
647-
!target.contains("freebsd") &&
648-
!target.contains("windows") &&
649-
!target.contains("apple") {
650-
let file = compiler_file(build,
651-
build.cxx(target).unwrap(),
652-
target,
653-
"libstdc++.a");
654-
cargo.env("LLVM_STATIC_STDCPP", file);
655-
}
656-
if build.config.llvm_link_shared {
657-
cargo.env("LLVM_LINK_SHARED", "1");
633+
match &*self.backend {
634+
"llvm" | "emscripten" => {
635+
// Build LLVM for our target. This will implicitly build the
636+
// host LLVM if necessary.
637+
let llvm_config = builder.ensure(native::Llvm {
638+
target,
639+
emscripten: self.backend == "emscripten",
640+
});
641+
642+
if self.backend == "emscripten" {
643+
features.push_str(" emscripten");
644+
}
645+
646+
let _folder = build.fold_output(|| format!("stage{}-rustc_trans", compiler.stage));
647+
println!("Building stage{} codegen artifacts ({} -> {}, {})",
648+
compiler.stage, &compiler.host, target, self.backend);
649+
650+
// Pass down configuration from the LLVM build into the build of
651+
// librustc_llvm and librustc_trans.
652+
if build.is_rust_llvm(target) {
653+
cargo.env("LLVM_RUSTLLVM", "1");
654+
}
655+
cargo.env("LLVM_CONFIG", &llvm_config);
656+
if self.backend != "emscripten" {
657+
let target_config = build.config.target_config.get(&target);
658+
if let Some(s) = target_config.and_then(|c| c.llvm_config.as_ref()) {
659+
cargo.env("CFG_LLVM_ROOT", s);
660+
}
661+
}
662+
// Building with a static libstdc++ is only supported on linux right now,
663+
// not for MSVC or macOS
664+
if build.config.llvm_static_stdcpp &&
665+
!target.contains("freebsd") &&
666+
!target.contains("windows") &&
667+
!target.contains("apple") {
668+
let file = compiler_file(build,
669+
build.cxx(target).unwrap(),
670+
target,
671+
"libstdc++.a");
672+
cargo.env("LLVM_STATIC_STDCPP", file);
673+
}
674+
if build.config.llvm_link_shared {
675+
cargo.env("LLVM_LINK_SHARED", "1");
676+
}
677+
}
678+
_ => panic!("unknown backend: {}", self.backend),
658679
}
659680

660-
run_cargo(build,
661-
&mut cargo,
662-
&librustc_trans_stamp(build, compiler, target),
663-
false);
681+
let tmp_stamp = build.cargo_out(compiler, Mode::Librustc, target)
682+
.join(".tmp.stamp");
683+
let files = run_cargo(build,
684+
cargo.arg("--features").arg(features),
685+
&tmp_stamp,
686+
false);
687+
let mut files = files.into_iter()
688+
.filter(|f| {
689+
let filename = f.file_name().unwrap().to_str().unwrap();
690+
is_dylib(filename) && filename.contains("rustc_trans-")
691+
});
692+
let codegen_backend = match files.next() {
693+
Some(f) => f,
694+
None => panic!("no dylibs built for codegen backend?"),
695+
};
696+
if let Some(f) = files.next() {
697+
panic!("codegen backend built two dylibs:\n{}\n{}",
698+
codegen_backend.display(),
699+
f.display());
700+
}
701+
let stamp = codegen_backend_stamp(build, compiler, target, self.backend);
702+
let codegen_backend = codegen_backend.to_str().unwrap();
703+
t!(t!(File::create(&stamp)).write_all(codegen_backend.as_bytes()));
664704
}
665705
}
666706

@@ -682,33 +722,29 @@ fn copy_codegen_backends_to_sysroot(builder: &Builder,
682722
// not linked into the main compiler by default but is rather dynamically
683723
// selected at runtime for inclusion.
684724
//
685-
// Here we're looking for the output dylib of the `RustcTrans` step and
725+
// Here we're looking for the output dylib of the `CodegenBackend` step and
686726
// we're copying that into the `codegen-backends` folder.
687727
let libdir = builder.sysroot_libdir(target_compiler, target);
688728
let dst = libdir.join("codegen-backends");
689729
t!(fs::create_dir_all(&dst));
690-
let stamp = librustc_trans_stamp(build, compiler, target);
691730

692-
let mut copied = None;
693-
for file in read_stamp_file(&stamp) {
694-
let filename = match file.file_name().and_then(|s| s.to_str()) {
695-
Some(s) => s,
696-
None => continue,
731+
for backend in builder.config.rust_codegen_backends.iter() {
732+
let stamp = codegen_backend_stamp(build, compiler, target, *backend);
733+
let mut dylib = String::new();
734+
t!(t!(File::open(&stamp)).read_to_string(&mut dylib));
735+
let file = Path::new(&dylib);
736+
let filename = file.file_name().unwrap().to_str().unwrap();
737+
// change `librustc_trans-xxxxxx.so` to `librustc_trans-llvm.so`
738+
let target_filename = {
739+
let dash = filename.find("-").unwrap();
740+
let dot = filename.find(".").unwrap();
741+
format!("{}-{}{}",
742+
&filename[..dash],
743+
backend,
744+
&filename[dot..])
697745
};
698-
if !is_dylib(filename) || !filename.contains("rustc_trans-") {
699-
continue
700-
}
701-
match copied {
702-
None => copied = Some(file.clone()),
703-
Some(ref s) => {
704-
panic!("copied two codegen backends:\n{}\n{}",
705-
s.display(),
706-
file.display());
707-
}
708-
}
709-
copy(&file, &dst.join(filename));
746+
copy(&file, &dst.join(target_filename));
710747
}
711-
assert!(copied.is_some(), "failed to find a codegen backend to copy");
712748
}
713749

714750
/// Cargo's output path for the standard library in a given stage, compiled
@@ -729,10 +765,12 @@ pub fn librustc_stamp(build: &Build, compiler: Compiler, target: Interned<String
729765
build.cargo_out(compiler, Mode::Librustc, target).join(".librustc.stamp")
730766
}
731767

732-
pub fn librustc_trans_stamp(build: &Build,
733-
compiler: Compiler,
734-
target: Interned<String>) -> PathBuf {
735-
build.cargo_out(compiler, Mode::Librustc, target).join(".librustc_trans.stamp")
768+
fn codegen_backend_stamp(build: &Build,
769+
compiler: Compiler,
770+
target: Interned<String>,
771+
backend: Interned<String>) -> PathBuf {
772+
build.cargo_out(compiler, Mode::Librustc, target)
773+
.join(format!(".librustc_trans-{}.stamp", backend))
736774
}
737775

738776
fn compiler_file(build: &Build,
@@ -849,10 +887,13 @@ impl Step for Assemble {
849887
compiler: build_compiler,
850888
target: target_compiler.host,
851889
});
852-
builder.ensure(RustcTrans {
853-
compiler: build_compiler,
854-
target: target_compiler.host,
855-
});
890+
for &backend in build.config.rust_codegen_backends.iter() {
891+
builder.ensure(CodegenBackend {
892+
compiler: build_compiler,
893+
target: target_compiler.host,
894+
backend,
895+
});
896+
}
856897
}
857898

858899
let stage = target_compiler.stage;
@@ -922,7 +963,9 @@ fn stderr_isatty() -> bool {
922963
}
923964
}
924965

925-
pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: bool) {
966+
pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: bool)
967+
-> Vec<PathBuf>
968+
{
926969
// Instruct Cargo to give us json messages on stdout, critically leaving
927970
// stderr as piped so we can get those pretty colors.
928971
cargo.arg("--message-format").arg("json")
@@ -1066,8 +1109,8 @@ pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: boo
10661109
let mut new_contents = Vec::new();
10671110
let mut max = None;
10681111
let mut max_path = None;
1069-
for dep in deps {
1070-
let mtime = mtime(&dep);
1112+
for dep in deps.iter() {
1113+
let mtime = mtime(dep);
10711114
if Some(mtime) > max {
10721115
max = Some(mtime);
10731116
max_path = Some(dep.clone());
@@ -1080,12 +1123,13 @@ pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: boo
10801123
if stamp_contents == new_contents && max <= stamp_mtime {
10811124
build.verbose(&format!("not updating {:?}; contents equal and {} <= {}",
10821125
stamp, max, stamp_mtime));
1083-
return
1126+
return deps
10841127
}
10851128
if max > stamp_mtime {
10861129
build.verbose(&format!("updating {:?} as {:?} changed", stamp, max_path));
10871130
} else {
10881131
build.verbose(&format!("updating {:?} as deps changed", stamp));
10891132
}
10901133
t!(t!(File::create(stamp)).write_all(&new_contents));
1134+
deps
10911135
}

0 commit comments

Comments
 (0)