Skip to content

Commit 21830b1

Browse files
authored
ci: Streamline CI browser playwright tests (#13276)
This PR streamlines our browser integration tests a bit more: 1. We now only install chromium for playwright tests in most places. This may help cut down test run time a bit, as we do not need to install native dependencies for webkit and firefox everywhere. 2. I changed how we split the matrix jobs for the browser integration tests. Previously, we sharded this via playwright. Now, we only do this for esm tests, which we now only run in chromium, not all browsers. Only the full bundle tests are run in all browsers (they are considerably faster than the esm tests...), and there we now do not use sharding, but have one matrix job for chromium (default), one for webkit and one for firefox (where each can only install the native dependencies it needs). 3. I renamed the jobs a bit so that we can see a bit more of the job name in the Github UI... it still truncates but you see more than before, at least... 4. For all other playwright tests (e.g. e2e tests etc) we only install chromium, AFAIK we do not have any non-chromium tests anywhere there. Some data points: * E2E tests seem to be about 10-20% faster than before, by looking through traces here: https://sentry.sentry.io/performance/trace/c5c1c643f1db0e8dd97df658405c2035/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&node=txn-8892bc908791478aba163dd3edd373b4&project=5899451&query=&sort=-timestamp&source=discover&statsPeriod=1h&timestamp=1723118364&yAxis=count%28%29 * For the playwright steps it's a bit harder to say as they are renamed, also they are not super consistent in execution time, we'll have to monitor it a bit afterwards, but subjectively it appears a bit shorter in most cases too.
1 parent adf7b40 commit 21830b1

File tree

2 files changed

+37
-18
lines changed

2 files changed

+37
-18
lines changed

.github/actions/install-playwright/action.yml

+6-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
name: "Install Playwright dependencies"
22
description: "Installs Playwright dependencies and caches them."
3+
inputs:
4+
browsers:
5+
description: 'What browsers to install.'
6+
default: 'chromium webkit firefox'
37

48
runs:
59
using: "composite"
@@ -17,12 +21,13 @@ runs:
1721
~/.cache/ms-playwright
1822
key: playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
1923

24+
# We always install all browsers, if uncached
2025
- name: Install Playwright dependencies (uncached)
2126
run: npx playwright install chromium webkit firefox --with-deps
2227
if: steps.playwright-cache.outputs.cache-hit != 'true'
2328
shell: bash
2429

2530
- name: Install Playwright system dependencies only (cached)
26-
run: npx playwright install-deps chromium webkit firefox
31+
run: npx playwright install-deps ${{ inputs.browsers || 'chromium webkit firefox' }}
2732
if: steps.playwright-cache.outputs.cache-hit == 'true'
2833
shell: bash

.github/workflows/build.yml

+31-17
Original file line numberDiff line numberDiff line change
@@ -528,7 +528,7 @@ jobs:
528528
run: yarn lerna run test --scope @sentry/profiling-node
529529

530530
job_browser_playwright_tests:
531-
name: Playwright (${{ matrix.bundle }}${{ matrix.shard && format(' {0}/{1}', matrix.shard, matrix.shards) || ''}}) Tests
531+
name: Playwright ${{ matrix.bundle }}${{ matrix.project && matrix.project != 'chromium' && format(' {0}', matrix.project) || ''}}${{ matrix.shard && format(' ({0}/{1})', matrix.shard, matrix.shards) || ''}} Tests
532532
needs: [job_get_metadata, job_build]
533533
if: needs.job_build.outputs.changed_browser_integration == 'true' || github.event_name != 'pull_request'
534534
runs-on: ubuntu-20.04-large-js
@@ -548,31 +548,30 @@ jobs:
548548
project:
549549
- chromium
550550
include:
551-
# Only check all projects for esm & full bundle
551+
# Only check all projects for full bundle
552552
# We also shard the tests as they take the longest
553553
- bundle: bundle_tracing_replay_feedback_min
554-
project: ''
555-
shard: 1
556-
shards: 2
554+
project: 'webkit'
557555
- bundle: bundle_tracing_replay_feedback_min
558-
project: ''
559-
shard: 2
560-
shards: 2
556+
project: 'firefox'
561557
- bundle: esm
562-
project: ''
558+
project: chromium
563559
shard: 1
564-
shards: 3
560+
shards: 4
565561
- bundle: esm
562+
project: chromium
566563
shard: 2
567-
shards: 3
564+
shards: 4
568565
- bundle: esm
569-
project: ''
566+
project: chromium
570567
shard: 3
571-
shards: 3
568+
shards: 4
569+
- bundle: esm
570+
project: chromium
571+
shard: 4
572+
shards: 4
572573
exclude:
573-
# Do not run the default chromium-only tests
574-
- bundle: bundle_tracing_replay_feedback_min
575-
project: 'chromium'
574+
# Do not run the un-sharded esm tests
576575
- bundle: esm
577576
project: 'chromium'
578577

@@ -592,12 +591,15 @@ jobs:
592591

593592
- name: Install Playwright
594593
uses: ./.github/actions/install-playwright
594+
with:
595+
browsers: ${{ matrix.project }}
595596

596597
- name: Run Playwright tests
597598
env:
598599
PW_BUNDLE: ${{ matrix.bundle }}
599600
working-directory: dev-packages/browser-integration-tests
600601
run: yarn test:ci${{ matrix.project && format(' --project={0}', matrix.project) || '' }}${{ matrix.shard && format(' --shard={0}/{1}', matrix.shard, matrix.shards) || '' }}
602+
601603
- name: Upload Playwright Traces
602604
uses: actions/upload-artifact@v3
603605
if: always()
@@ -606,7 +608,7 @@ jobs:
606608
path: dev-packages/browser-integration-tests/test-results
607609

608610
job_browser_loader_tests:
609-
name: Playwright Loader (${{ matrix.bundle }}) Tests
611+
name: PW ${{ matrix.bundle }} Tests
610612
needs: [job_get_metadata, job_build]
611613
if: needs.job_build.outputs.changed_browser_integration == 'true' || github.event_name != 'pull_request'
612614
runs-on: ubuntu-20.04
@@ -639,6 +641,8 @@ jobs:
639641

640642
- name: Install Playwright
641643
uses: ./.github/actions/install-playwright
644+
with:
645+
browsers: chromium
642646

643647
- name: Run Playwright Loader tests
644648
env:
@@ -750,8 +754,12 @@ jobs:
750754
uses: ./.github/actions/restore-cache
751755
env:
752756
DEPENDENCY_CACHE_KEY: ${{ needs.job_build.outputs.dependency_cache_key }}
757+
753758
- name: Install Playwright
754759
uses: ./.github/actions/install-playwright
760+
with:
761+
browsers: chromium
762+
755763
- name: Run integration tests
756764
env:
757765
NODE_VERSION: ${{ matrix.node }}
@@ -953,6 +961,8 @@ jobs:
953961

954962
- name: Install Playwright
955963
uses: ./.github/actions/install-playwright
964+
with:
965+
browsers: chromium
956966

957967
- name: Get node version
958968
id: versions
@@ -1050,6 +1060,8 @@ jobs:
10501060

10511061
- name: Install Playwright
10521062
uses: ./.github/actions/install-playwright
1063+
with:
1064+
browsers: chromium
10531065

10541066
- name: Get node version
10551067
id: versions
@@ -1150,6 +1162,8 @@ jobs:
11501162

11511163
- name: Install Playwright
11521164
uses: ./.github/actions/install-playwright
1165+
with:
1166+
browsers: chromium
11531167

11541168
- name: Get node version
11551169
id: versions

0 commit comments

Comments
 (0)