Compiling rust from source uses all available threads

[sjackman]

When building rust from source it uses all available threads. How do I specify the number of threads?

Building rust from source on CircleCI fails because it uses 32 threads and exceeds the available memory. I use make -j12 but it uses 32 threads nonetheless.
There are 32 concurrent processes of cc1plus in this dump from top https://4264-56942570-gh.circle-artifacts.com/0/tmp/memory-usage.txt
See the log file https://circleci.com/gh/Linuxbrew/homebrew-core/4264
and the original issue https://github.com/Linuxbrew/homebrew-core/pull/1953

[alexcrichton]

Oh this happened due to the switch to a new build system on our end. The build system has a makefile but it's just a thin veneer over the actual build system, what we call rustbuild.

The entry point of the build system, the x.py script in the root of the repository, supports a -j option like make which should do the trick here. Unfortunately it doesn't look like there's an easy way for us to infer the -j argument passed to make itself, so the best solution for now may be to update to replacing your invocation of make with ./x.py build -j12. Would that work?

[sjackman]

Hi, Alex. Thanks for your quick reply.

I expected that the number of requested threads could be parsed out of the environment variable MAKEFLAGS, but sadly no.

❯❯❯ echo 'all: ; echo $(MAKEFLAGS)' | make -f /dev/stdin -j8
echo  --jobserver-fds=3,4 -j
--jobserver-fds=3,4 -j

so the best solution for now may be to update to replacing your invocation of make with ./x.py build -j12. Would that work?

Yep, that works just fine for me. So I run configure as before and then run ./x.py build -j12?

[sjackman]

The file https://static.rust-lang.org/dist/rustc-1.15.1-src.tar.gz doesn't contain ./x.py. What should I do when building using that source tarball?

[alexcrichton]

Oh right sorry that was actually a bug in the 1.15.1 tarball :(.

For now on 1.15 you can pass --disable-rustbuild to configure and then make -jN should work correctly as before (it uses the old build system). For 1.16+ you should be able to use the x.py trick.

And yeah you'll still want to use ./configure ahead of time.

[retep998]

Alternatively, instead of using ./configure you can create a config.toml based off src/bootstrap/config.toml.example and configure your build that way. It's definitely more flexible.

[Mark-Simulacrum]

I believe there's not a bug here; closing.