build: Update Lerna to v6 and use Nx caching for builds #6555

mydea · 2022-12-15T14:05:23Z

Update Lerna to v6, which uses Nx under the hood.

Impact

For now, we only use nx for caching of build steps. We can explore, in the future, also using further nx functionality, like running only changed tasks etc.

But even now, it can cut down time of the build step by about 6 minutes. Overall test run times have been ~10-11 minutes for me now.

Streamline package scripts

In order to setup the caching etc. reasonably, I unified the build scripts a bit. Now, there are four main scripts:

build:types
build:bundle
build:tarball (formerly build:npm) - IMHO describes better what this does, it wasn't super clear to me before
build:transpile (formerly build:rollup) - to make this more generic, e.g. for angular

With these, we can do a better generic task dependency graph, as we can always depend on the same types of tasks.

Note that we do not need any dev commands on the top level anymore really, as with caching that shouldn't be that necessary anymore.

Note there is also an build:transpile:uncached command that can be specified by packaged for things that should never be cached. This is currently only used for utils, which generates symlinks which are not cached nicely.

I got rid of the build:extras stuff, which is not needed anymore with this setup.

Node unit tests

Node unit tests have already been split out in a previous task from browser unit test.
The challenge is that Lerna 6 requires node >=14. So for older node versions, we install lerna 3 again.

Status

There is def. some more cleanup to be made with remaining tasks etc. But we can already check it out.

github-actions · 2022-12-15T14:56:37Z

size-limit report 📦

Path	Size
@sentry/browser - ES5 CDN Bundle (gzipped + minified)	19.84 KB (+0.01% 🔺)
@sentry/browser - ES5 CDN Bundle (minified)	61.46 KB (0%)
@sentry/browser - ES6 CDN Bundle (gzipped + minified)	18.62 KB (+0.02% 🔺)
@sentry/browser - ES6 CDN Bundle (minified)	55 KB (0%)
@sentry/browser - Webpack (gzipped + minified)	20.38 KB (0%)
@sentry/browser - Webpack (minified)	66.55 KB (0%)
@sentry/react - Webpack (gzipped + minified)	20.4 KB (0%)
@sentry/nextjs Client - Webpack (gzipped + minified)	47.63 KB (0%)
@sentry/browser + @sentry/tracing - ES5 CDN Bundle (gzipped + minified)	26.82 KB (+0.03% 🔺)
@sentry/browser + @sentry/tracing - ES6 CDN Bundle (gzipped + minified)	25.25 KB (-0.01% 🔽)
@sentry/replay ES6 CDN Bundle (gzipped + minified)	43.05 KB (+0.01% 🔺)
@sentry/replay - Webpack (gzipped + minified)	38.3 KB (0%)

lforst · 2022-12-16T09:08:22Z

Caching can be a bit of a footgun in my experience. Wdyt about not using any cache for master/release branch CI?

Lms24

Looking good so far. Would be great to get the NX goodness + speed increase!

A couple of questions:

build:pack (formerly build:rollup) - to make this more generic, e.g. for angular

I know this is a nit but IMO pack doesn't really describe this step well. IIUC what we're doing in this step is transpiling with rollup or "compiling" with other compilers/bundlers (ngc in Angular for instance). I know it isn't a perfect name either but wdyt about build:transpile?

Note that we do not need any dev commands on the top level anymore really, as with caching that shouldn't be that necessary anymore.

So the idea is to simply run build:pack instead of build:dev? (I guess this works, as long as build:pack depends on build:types)

Great change, would love to get this in!
Let's just make sure that before we merge this, we create a fake release branch and test artifact upload. I just want to make sure that we're still creating all bundles and tarballs as before.

Lms24 · 2022-12-16T09:49:48Z

nx.json

+    "build:pack": {
+      "dependsOn": [
+        "build:types",
+        "build:pack:uncached",
+        "^build:pack",
+        "^build:types",
+        "^build:pack:uncached"


Q: Why does build:pack (which iiuc is our transpiling step) depend on type checking? IIRC we purposefully ran them in parallel before, as we'd still get TS errors via build:types.

BTW I'm not saying we should change this again, was just curious.

I guess ideally we can get rid of this parallel building (I'd say that nx should handle that fine) and just make use of dependent steps properly. So actually we only run build:bundle or build:pack and get everything we need (including the types)?

Lms24 · 2022-12-16T09:52:32Z

nx.json

+      "build:bundle": {
+        "dependsOn": [],


Q: should this also depend on ^build:types?

Probably, I guess! I'll add it.

mydea · 2023-01-05T08:42:08Z

Caching can be a bit of a footgun in my experience. Wdyt about not using any cache for master/release branch CI?

yeah, good point! I guess we can disable it in those cases, should not be too hard, I'll look into it.

I know this is a nit but IMO pack doesn't really describe this step well. IIUC what we're doing in this step is transpiling with rollup or "compiling" with other compilers/bundlers (ngc in Angular for instance). I know it isn't a perfect name either but wdyt about build:transpile?

Yeah, I was also not super happy with build:pack, but couldn't think of something better 🤔 IMHO transpile is a bit annoying to type/write (easy to typo 😅 ). Maybe something like build:output or build:prepare?

Or, another alternative, make it build:bundle (former build:rollup) and build:bundle-cdn?

So the idea is to simply run build:pack instead of build:dev? (I guess this works, as long as build:pack depends on build:types)

I guess we can also simply leave build:dev which does not build cdn bundles. Will still be faster than caching 😅

mydea · 2023-01-05T09:00:52Z

.github/workflows/build.yml

+        # - when PR has `ci-skip-cache` label
+        if:
+          github.ref != 'refs/heads/master' && !startsWith(github.ref, 'refs/heads/release/') &&
+          !github.event.pull_request.labels.contains('ci-skip-cache')


The idea is you can add a label ci-skip-cache to the PR, which ensures it will not do any nx caching. May be helpfup when debugging here or there (maybe we could even extend this to the other caches, like the dependency cache?)

lforst · 2023-01-11T09:23:50Z

lerna.json

  "npmClient": "yarn",
-  "useWorkspaces": true
+  "useWorkspaces": true,
+  "$schema": "node_modules/lerna/schemas/lerna-schema.json"


giga nit: can we move this line to the top of the json :D

Lms24

So happy about this change! Thanks for tackling this.

AbhiPrasad

🚀

JamesHenry · 2023-02-03T10:37:32Z

Hi Sentry Folks 👋

I'm James, you might possibly know me already from having created https://github.com/typescript-eslint/typescript-eslint or my work on some other popular open-source tools over the years such as ESLint, Prettier and Babel. Right now I've got my Lerna maintainer hat on 🐉

I was delighted to see this PR via Jay's tweet and that you have already pressed ahead with maximizing the benefits of Lerna's modern task-runner by enabling caching.

I was the 5th employee of Nx (formerly known as Nrwl) back in 2017 and it's been so rewarding to be able build that tool as open-sourced, MIT licensed software from the very beginning. Now that it's even able to benefit other tools like Lerna, that's the cherry on top!

Since our team took over stewardship of Lerna in the summer last year, I have been responsible for it, and initially that meant steadying the ship and making sure the repository got some consistent love.

I'm super excited about this next phase, however, in which I can now start reaching out to other great open-source teams like yours and gather any feedback you may have and/or assist you with getting the most out of it.

With that in mind, I am going to spend some time today playing with your repo to see if there are any optimizations I can suggest.

Another totally optional thing - if you have a paid team Slack account, we would also be delighted to leverage Slack Connect between our workspaces to allow for frictionless ongoing communication. If that's something you are interested in, please send me a DM with your relevant Slack details: https://twitter.com/MrJamesHenry

If you have a similar chat tool you would like me to join instead, then I am definitely open to that.

Cheers!

James

mydea self-assigned this Dec 15, 2022

mydea force-pushed the fn/lerna-6 branch from bbb199b to ade9f8f Compare December 15, 2022 14:17

mydea force-pushed the fn/lerna-6 branch 2 times, most recently from 79c50b6 to 9c709bf Compare December 15, 2022 17:00

mydea changed the title ~~[WIP] Try lerna with nx~~ build: Update Lerna to v6 and use Nx caching for builds Dec 16, 2022

Lms24 reviewed Dec 16, 2022

View reviewed changes

mydea added the Status: Backlog label Jan 5, 2023

mydea force-pushed the fn/lerna-6 branch from 733e5ae to 8d38995 Compare January 5, 2023 09:00

mydea commented Jan 5, 2023

View reviewed changes

mydea force-pushed the fn/lerna-6 branch from a17f4f9 to 50872cc Compare January 5, 2023 10:35

mydea added the ci-skip-cache label Jan 9, 2023

mydea force-pushed the fn/lerna-6 branch 2 times, most recently from 073856d to f8d811d Compare January 10, 2023 13:31

mydea marked this pull request as ready for review January 10, 2023 13:34

mydea requested review from AbhiPrasad and lforst January 10, 2023 13:34

mydea removed the ci-skip-cache label Jan 10, 2023

mydea force-pushed the fn/lerna-6 branch from f8d811d to 8b6655b Compare January 10, 2023 16:26

mydea added 4 commits January 11, 2023 10:14

build: Update to lerna 6 & use nx cache for build

a383511

ref: Ensure we always build when skipping cache

57613ab

ref: Rename build:pack to build:transpile

3d41594

fix typo

ca6fe0a

mydea force-pushed the fn/lerna-6 branch from eebc177 to ca6fe0a Compare January 11, 2023 09:14

lforst reviewed Jan 11, 2023

View reviewed changes

PR fixes

3c9a876

Lms24 approved these changes Jan 11, 2023

View reviewed changes

lforst approved these changes Jan 11, 2023

View reviewed changes

PR fixes

5338268

AbhiPrasad approved these changes Jan 11, 2023

View reviewed changes

fix

22a1277

mydea merged commit 95ba49a into master Jan 11, 2023

mydea deleted the fn/lerna-6 branch January 11, 2023 10:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

build: Update Lerna to v6 and use Nx caching for builds #6555

build: Update Lerna to v6 and use Nx caching for builds #6555

Uh oh!

mydea commented Dec 15, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Dec 15, 2022 •

edited

Loading

Uh oh!

lforst commented Dec 16, 2022

Uh oh!

Lms24 left a comment

Uh oh!

Lms24 Dec 16, 2022

Uh oh!

mydea Jan 5, 2023

Uh oh!

Lms24 Dec 16, 2022

Uh oh!

mydea Jan 5, 2023

Uh oh!

mydea commented Jan 5, 2023

Uh oh!

mydea Jan 5, 2023

Uh oh!

lforst Jan 11, 2023

Uh oh!

Lms24 left a comment

Uh oh!

AbhiPrasad left a comment

Uh oh!

JamesHenry commented Feb 3, 2023

Uh oh!

Uh oh!

Uh oh!

build: Update Lerna to v6 and use Nx caching for builds #6555

build: Update Lerna to v6 and use Nx caching for builds #6555

Uh oh!

Conversation

mydea commented Dec 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Impact

Streamline package scripts

Node unit tests

Status

Uh oh!

github-actions bot commented Dec 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

size-limit report 📦

Uh oh!

lforst commented Dec 16, 2022

Uh oh!

Lms24 left a comment

Choose a reason for hiding this comment

Uh oh!

Lms24 Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

mydea Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

Lms24 Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

mydea Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

mydea commented Jan 5, 2023

Uh oh!

mydea Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

lforst Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

Lms24 left a comment

Choose a reason for hiding this comment

Uh oh!

AbhiPrasad left a comment

Choose a reason for hiding this comment

Uh oh!

JamesHenry commented Feb 3, 2023

Uh oh!

Uh oh!

mydea commented Dec 15, 2022 •

edited

Loading

github-actions bot commented Dec 15, 2022 •

edited

Loading