-
Notifications
You must be signed in to change notification settings - Fork 13.3k
Add enzyme distribution step #140244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Shourya742
wants to merge
4
commits into
rust-lang:master
Choose a base branch
from
Shourya742:2025-04-24-enzyme-distribution
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+69
−3
Open
Add enzyme distribution step #140244
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, this is a weird. When I have
download-ci-llvm = true
enabled, this LLVM config is the CI LLVM (which is version 20). But when I actually build Enzyme locally, it generateslibEnzyme-18
. My host system LLVM is LLVM 18, so that looks related? I don't know why is the filename containing the version of the host compiler rather than the version of the LLVM source code though.@ZuseZ4 Does Enzyme have its own versioning separate from LLVM? I found some version
0.0.79
in the source code.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rg "0\.0\.79"
doesn't return anything in Enzyme, where did you find it?Enzyme has releases but doesn't follow semver (
in a meaningful wayonly uses patch versions), so I conveniently ignore it.https://github.com/EnzymeAD/Enzyme/releases
We also just use Enzyme's LLVM Pass which allows us to ignore most of the API, so breaking changes barely ever affect us.
They are more relevant for Julia which has a different build infra working on gh releases. We just use the source code from rust-lang/Enzyme.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just for clarification of
I currently only try (and have it working):
A) building LLVM locally, then building Enzyme based on it.
Here we try:
B) building LLVM remote (in CI), then building Enzyme also remote based on it.
I guess you tried
C) building LLVM remote, then downloading it and building Enzyme locally against it?
I thought based on Zulip that you wanted a different solution for C, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Semver is followed for user-level enzyme_autodiff and related functions. All user-level changes have been backwards compatible, so we've never bumped. Internal APIs (like LLVM) have no guarantees of stability
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I saw the version here: https://github.com/rust-lang/enzyme/blob/main/enzyme/CMakeLists.txt#L10
To clarify, I was indeed using option C), but almost inadvertedly, as
download-ci-llvm
is enabled in a lot of default configs and many people use it, it would be nice if bootstrap either "just worked" with it, or at least if it produced a loud error.I don't really understand how this works though. The build step for Enzyme seems to pass
LLVM_CONFIG_REAL
to thellvm-config
of the LLVM downloaded from CI. I would thus expect that Enzyme either attaches to that, or that it is somehow linked to the in-tree version of LLVM. But instead it seems to produce a version number tied to the host LLVM used to build Enzyme itself.How exactly is Enzyme "tied" to a specific LLVM sysroot/source code/build/whatever?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Enzyme works with the LLVM it was build against, everything else could only work by coincidence. E.g. this is one of the cmake commands I use locally:
cmake .. -G Ninja -DLLVM_DIR=/home/manuel/prog/rust-middle/build/x86_64-unknown-linux-gnu/llvm/build/lib/cmake/llvm -DLLVM_EXTERNAL_LIT=/home/manuel/prog/rust-middle/src/llvm-project/llvm/utils/lit/lit.py -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=YES -DBUILD_SHARED_LIBS=On
We should build Enzyme wherever we also build LLVM. So after building LLVM, we would build Enzyme and set
LLVM_DIR
to the path of the newly build llvm. We should not point to the LLVM/Clang that was used to build Enzyme and LLVM.I mean that's why I added the
-DLLVM_DIR
definition in rust/src/bootstrap/src/core/build_steps/llvm.rs, it just looks like we don't set it correctly in CI right now? So I guessbuilder.llvm_out(target)
is wrong then (?)As another puzzle piece, these folders from an apple CI run seem correct? We build using LLVM/Clang-15, but the self-build LLVM: https://github.com/rust-lang-ci/rust/actions/runs/14582626710/job/40902344579
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I see now, I didn't realize that
LLVM_DIR
is the thing that links Enzyme to LLVM, I thought it's the output directory where the Enzyme artifacts will be generated.In that case I suppose that without
download-ci-llvm
it should mostly do what we want. I think that the path we give toLLVM_DIR
is not exactly what it should be, as it's essentially justbuild/<target>/llvm
; it looks like Enzyme's CMake expects it to be something a bit different (e.g. the path you use,build/<target>/llvm/build/lib/cmake/llvm
. That being said, the Enzyme CMake seems to use recursive greps for finding the files that it needs, so it looks like it is able to resolve what it requires.Ok, in that case I'm fine with just adding an error into the
llvm::Enzyme
step that it currently cannot be combined withdownload-ci-llvm
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, in case that the code isn't clear, @Shourya742 can you please add a comment around line 973, saying that we build enzyme against this specific LLVM (and using it with a different LLVM isn't supported)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also @Kobzol now that I already have you here, do you mind looking at the apple issue here? :D https://github.com/rust-lang-ci/rust/actions/runs/14582626710/job/40902344579#step:28:28938 and checking if the llvm path looks right to you? I feel like it should be ok, 15 is used for building everything, 20 is the one we build against. But it still fails to link LLVM. Presumably Enzyme's cmake should not just link against llvm (-lLLVM), but also specify
-L
based on the given LLVM_DIR?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not exactly sure how linking works on Apple, but just passing
-lLLVM
is unlikely to work, IMO. The directory where thelibLLVM.dylib
(or static archive) files can be found will likely be needed.