Skip to content

Bump tokenizers dep #10332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 25, 2025
Merged

Bump tokenizers dep #10332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a28bcc9
Try recursive
jackzhxng Apr 25, 2025
39be387
Bump tokenizers dep
jackzhxng Apr 21, 2025
2520c74
Shim?
jackzhxng Apr 21, 2025
ed13c16
Lint
jackzhxng Apr 24, 2025
20517e4
Update to main again
jackzhxng Apr 24, 2025
49c2c2c
Sync again
jackzhxng Apr 24, 2025
ac80d7b
TRY PATCH
jackzhxng Apr 25, 2025
b388625
See wtf is in the gh action machine
jackzhxng Apr 25, 2025
787b9b4
Remove prints
jackzhxng Apr 25, 2025
8476540
Update tokenizers version again
jackzhxng Apr 25, 2025
704bfbd
Update tokenizers to main
jackzhxng Apr 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/_unittest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: ${{ inputs.docker-image }}
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -45,7 +45,7 @@ jobs:
with:
runner: macos-m1-stable
python-version: '3.11'
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
script: |
set -eux
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/android-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ jobs:
with:
runner: linux.2xlarge.memory
docker-image: executorch-ubuntu-22.04-qnn-sdk
submodules: 'true'
submodules: 'recursive'
timeout: 60
upload-artifact: android-models
upload-artifact-to-s3: true
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/apple-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ jobs:
# NB: Need to use our AWS MacOS runner to upload large models to S3
runner: macos-m1-stable
python-version: '3.11'
submodules: 'true'
submodules: 'recursive'
timeout: 60
upload-artifact: ios-models
upload-artifact-to-s3: true
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/doc-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
job-name: Build doc
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12-android
submodules: 'true'
submodules: 'recursive'
repository: pytorch/executorch
upload-artifact: docs
timeout: 90
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-linter
submodules: 'true'
submodules: 'recursive'
fetch-depth: 0
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/periodic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
with:
runner: ${{ matrix.runner }}
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: ${{ matrix.timeout }}
script: |
Expand Down
46 changes: 23 additions & 23 deletions .github/workflows/pull.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-gcc9
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -66,7 +66,7 @@ jobs:
with:
runner: ${{ matrix.runner }}
docker-image: ${{ matrix.docker-image }}
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -119,7 +119,7 @@ jobs:
with:
runner: ${{ matrix.runner }}
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -165,7 +165,7 @@ jobs:
with:
runner: ${{ matrix.runner }}
docker-image: ${{ matrix.docker-image }}
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 900
script: |
Expand Down Expand Up @@ -197,7 +197,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12-android
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -222,7 +222,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -246,7 +246,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -270,7 +270,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -301,7 +301,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -331,7 +331,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -354,7 +354,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -382,7 +382,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-gcc9
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -418,7 +418,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -489,7 +489,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-arm-sdk
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down Expand Up @@ -528,7 +528,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-qnn-sdk
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 900
script: |
Expand Down Expand Up @@ -563,7 +563,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-qnn-sdk
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 180
script: |
Expand Down Expand Up @@ -596,7 +596,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-qnn-sdk
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 180
script: |
Expand All @@ -618,7 +618,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -645,7 +645,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -672,7 +672,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -699,7 +699,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-clang12
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -726,7 +726,7 @@ jobs:
with:
runner: linux.24xlarge
docker-image: executorch-ubuntu-22.04-mediatek-sdk
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand All @@ -747,7 +747,7 @@ jobs:
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-gcc9
submodules: 'true'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
Expand Down
Loading
Loading