-
Notifications
You must be signed in to change notification settings - Fork 13.6k
Add pre-merge workflow for HLSL testing #122184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 11 commits
a6c0375
b2cee9e
e7372f7
12f924f
7c9763a
4dd8503
22fa02c
7c9f41f
a8b3089
c8fce72
18a274f
8d155e1
9969eea
b0bbdff
f165e31
bd85a5e
ce1b8b0
583f466
d85eb92
035b1d8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
name: HLSL Tests - macOS | ||
|
||
permissions: | ||
contents: read | ||
|
||
on: | ||
workflow_dispatch: | ||
pull_request: | ||
branches: | ||
- main | ||
paths: | ||
- llvm/**/DirectX/** | ||
- .github/workflows/hlsl* | ||
- clang/*HLSL*/**/* | ||
- clang/**/*HLSL* | ||
- llvm/**/Frontend/HLSL/**/* | ||
|
||
jobs: | ||
HLSL-macOS: | ||
uses: ./.github/workflows/hlsl-test-all.yaml | ||
with: | ||
SKU: hlsl-macos | ||
TestTarget: check-hlsl-clang-mtl | ||
LLVM-branch: ${{ github.ref }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,157 @@ | ||
name: HLSL Test | ||
|
||
permissions: | ||
contents: read | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
LLVM-Ref: | ||
description: 'Test Suite Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
LLVM-branch: | ||
description: 'LLVM Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
DXC-branch: | ||
description: 'DXC Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
BuildType: | ||
description: 'Build Type' | ||
required: false | ||
default: 'Release' | ||
type: choice | ||
options: | ||
- Release | ||
- RelWithDebInfo | ||
- Debug | ||
TestTarget: | ||
required: false | ||
default: 'check-hlsl' | ||
type: string | ||
Test-Clang: | ||
required: true | ||
type: choice | ||
options: | ||
- On | ||
- Off | ||
SKU: | ||
required: true | ||
type: choice | ||
options: | ||
- hlsl-metal | ||
LLVM-ExtraCMakeArgs: | ||
description: 'Extra CMake Args for LLVM' | ||
required: false | ||
default: '' | ||
type: string | ||
workflow_call: | ||
inputs: | ||
OffloadTest-branch: | ||
description: 'Test Suite Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
LLVM-branch: | ||
description: 'LLVM Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
DXC-branch: | ||
description: 'DXC Branch' | ||
required: false | ||
default: 'main' | ||
type: string | ||
SKU: | ||
required: true | ||
type: string | ||
BuildType: | ||
description: 'Build Type' | ||
required: false | ||
default: 'Release' | ||
type: string | ||
Test-Clang: | ||
required: false | ||
default: 'On' | ||
type: string | ||
TestTarget: | ||
required: false | ||
default: 'check-hlsl' | ||
type: string | ||
LLVM-ExtraCMakeArgs: | ||
description: 'Extra CMake Args for LLVM' | ||
required: false | ||
default: '' | ||
type: string | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a lot of inputs do you really need all of these? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can strip down and simplify what is in the LLVM tree. They exist because they have all been used in the dispatch workflow on the test suite repo. It's useful for running one-off tests. The workflow we use for LLVM pre-merge doesn't need that so I'll strip them down. |
||
|
||
jobs: | ||
build: | ||
runs-on: [self-hosted, "${{ inputs.SKU }}"] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm fairly certain you can remove the |
||
steps: | ||
- name: Checkout DXC | ||
uses: actions/checkout@v4 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For this and all the actions we use, it should be pinned to a specific git hash and not use labels. See https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions |
||
with: | ||
repository: Microsoft/DirectXShaderCompiler | ||
ref: ${{ inputs.DXC-branch }} | ||
path: DXC | ||
fetch-depth: 1 | ||
submodules: true | ||
- name: Checkout LLVM | ||
uses: actions/checkout@v4 | ||
with: | ||
repository: llvm/llvm-project | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should be $GITHUB_RESPOSITORY, which is the default, so you can actually just omit this. |
||
ref: ${{ inputs.LLVM-branch }} | ||
path: llvm-project | ||
fetch-depth: 1 | ||
- name: Checkout OffloadTest | ||
uses: actions/checkout@v4 | ||
with: | ||
repository: llvm-beanz/offload-test-suite | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm assuming this is the repository for https://discourse.llvm.org/t/rfc-proposal-for-offload-execution-test-suite/83947? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. |
||
ref: ${{ inputs.OffloadTest-branch }} | ||
path: OffloadTest | ||
fetch-depth: 1 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe this is the default and can probably be removed? |
||
- name: Checkout Golden Images | ||
uses: actions/checkout@v4 | ||
with: | ||
repository: llvm-beanz/offload-golden-images | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How are these images generated? Is there a reason this needs to be a separate repository? Are they expected to stay completely static? When working with something like this in the past I shipped images as a Github release, but I would had to regenerate things version to version of the software that I was trying to support. If things are expected to remain static, keeping them in a git repo seems reasonable enough to me. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They are expected to mostly remain static, but also expected to be significant in size as we flesh out the test coverage. They're in a separate repository from the other tests because they are optional. Most tests do not rely on image comparison for correctness. |
||
ref: main | ||
path: golden-images | ||
fetch-depth: 1 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here. |
||
- name: Setup Windows | ||
if: runner.os == 'windows' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This needs to be capitalized: |
||
uses: llvm/actions/setup-windows@main | ||
with: | ||
arch: amd64 | ||
- name: Build DXC | ||
run: | | ||
cd DXC | ||
mkdir build | ||
cd build | ||
cmake -G Ninja -DCMAKE_BUILD_TYPE=${{ inputs.BuildType }} -C ${{ github.workspace }}/DXC/cmake/caches/PredefinedParams.cmake -C ${{ github.workspace }}/OffloadTest/cmake/caches/sccache.cmake -DHLSL_DISABLE_SOURCE_GENERATION=On ${{ github.workspace }}/DXC/ | ||
ninja | ||
- name: Build LLVM | ||
run: | | ||
cd llvm-project | ||
mkdir build | ||
cd build | ||
cmake -G Ninja ${{ inputs.LLVM-ExtraCMakeArgs }} -DDXIL_DIS=${{ github.workspace }}/DXC/build/bin/llvm-dis -DLLVM_INCLUDE_DXIL_TESTS=On -DCMAKE_BUILD_TYPE=${{ inputs.BuildType }} -C ${{ github.workspace }}/llvm-project/clang/cmake/caches/HLSL.cmake -C ${{ github.workspace }}/OffloadTest/cmake/caches/sccache.cmake -DDXC_DIR=${{ github.workspace }}/DXC/build/bin -DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=${{ github.workspace }}/OffloadTest -DLLVM_EXTERNAL_PROJECTS="OffloadTest" -DLLVM_LIT_ARGS="--xunit-xml-output=testresults.xunit.xml -v" -DOFFLOADTEST_TEST_CLANG=${{ inputs.Test-Clang }} -DGOLDENIMAGE_DIR=${{ github.workspace }}/golden-images ${{ github.workspace }}/llvm-project/llvm/ | ||
ninja hlsl-test-depends llvm-test-depends clang-test-depends | ||
- name: Run HLSL Tests | ||
run: | | ||
cd llvm-project | ||
cd build | ||
ninja check-llvm | ||
ninja check-clang | ||
ninja check-hlsl-unit | ||
ninja ${{ inputs.TestTarget }} | ||
- name: Publish Test Results | ||
uses: EnricoMi/publish-unit-test-result-action/macos@v2 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems like a reasonable enough first approach for publishing test results to me. We need to figure out something like this for the main premerge pipeline and I'm not exactly sure what we want to do... Good to know that this exists. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's a bit janky... I don't love it. I'm really annoyed that GitHub doesn't have first class support for test result publication. It is better than nothing 😢 . |
||
if: always() && runner.os == 'macOS' | ||
with: | ||
comment_mode: off | ||
files: llvm-project/build/**/testresults.xunit.xml |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the structure of these files is not going to scale well. The way you have it structured, you will need to add a new file for each OS and the files will be mostly the same. I think what you want is something more like this: https://github.com/llvm/llvm-project/blob/main/.github/workflows/release-binaries-all.yml#L73
Where you have the hlsl-test-all.yaml file with a job that has a matrix of all the supported operating systems, and then each matrix entry calls hlsl-test.yaml (which is what you now have in hlsl-test-all.yml) to run the actual tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, that makes sense. The structuring as it is was to produce status badges:

(see: https://github.com/llvm-beanz/offload-test-suite/?tab=readme-ov-file#current-status)
Those require unique workflows at the top level and can't provide matrix granularity.
Obviously this isn't a consideration here, so I can refactor this.