Distribute a prebuilt swift-syntax binary

[petrpavlik]

Description

I think that it'd be great to also distribute swift-syntax as a prebuilt framework with every release, at least for Apple platforms (like https://github.com/realm/SwiftLint does).

We have been successfully experimenting with https://github.com/sjavora/swift-syntax-xcframeworks because the added build time to our CI builds when compiling swift-syntax from the source code is unacceptable.

[ahoppen]

Tracked in Apple’s issue tracker as rdar://120712265

[kingnight]

Strongly agree with this suggestion

[asam139]

Tracked in Apple’s issue tracker as rdar://120712265

@ahoppen can you give an estimation? For huge projects the initial compile time is a huge factor so it can be integrated

[vmanot]

@ahoppen would love to know if there's any estimation available regarding a fix, this is a showstopper for our team! 🙏

[ahoppen]

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

[vmanot]

@ahoppen given that Package.swift supports platform conditionals (i.e. #if os(macOS)), I'd imagine that a temporary patch for Apple platforms (perhaps one pointing to XCFrameworks just for Apple platform builds) would unblock a lot of teams on adopting macros and immediately free up developers from a massive amount of pain experienced on a daily-basis right now.

This issue has been plaguing adopters since the release of Swift Macros. I completely understand and fully support the desire for a fix that supports Linux and Windows, but my understanding is that SwiftPM's current support for XCFrameworks is capable of solving this issue today for Apple platform developers, in a conditional manner that will not disrupt Windows/Linux builds.

There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).

Simply adding SwiftSyntax as a dependency can increase Xcode Cloud build times by up to 12 minutes. It has absolute wrecked release build times, and Xcode's inability to reuse build products for even unchanging dependencies across schemes/branches means that this cost is incurred over and over again during a typical work session with a moderately complex project that adopts macros.

Furthermore, Xcode 15's accruing instability necessitates frequently cleaning DerivedData to fix ghost errors/warnings across projects, and Xcode currently does not offer an option to clean the derived data while preserving build products/artifacts for fixed dependencies. This means that swift-syntax and its dependencies is rebuilt in debug mode multiple times as well. This isn't an issue with the swift-syntax project of course, but it is a practical consideration given that a large majority of folks using swift-syntax right now are only using it because it is a practical requirement for Swift macro targets.

[vsarunas]

Is there any plans to address this for Swift 6?

[vmanot]

@ahoppen would love to know if there are any updates on this following all the incredible announcements from WWDC24! 🙏

[flockoffiles]

We would also very much want to have SwiftSyntax provided as an xcframework (since we cannot link directly to GitHub for the sources). We are building it ourselves now with swift-create-xcframework.

[leekurg]

Any updates? Waiting for solution for half of a year, struggling with 400% increased build time for medium sized project on day-to-day basis.

[cameroncooke]

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

The most frustrating thing about Swift these days is the ideology that Swift will be one language to rule them all across all platforms when in reality not many outside of the Apple platform community care. This constant unwillingness to support Apple platform-only enhancements punishes 99% of the community that uses Swift, which are developers on Apple platforms.

[vmanot]

@cameroncooke I echo your sentiment, but I don't think it's productive to imply or state that this is unwillingness. Moreover, I have many reasons to believe the opposite, and I'm confident that it is definitely some other reason (which, of course, is unbeknown to us, and that in itself is frustrating and one of the major issues here).

A few points I'd like to note:

1. Many, many, many folks have echoed that they encounter this issue daily, and it is nontrivial in how it impacts their build times.
2. Even after WWDC24, which focused on exciting new developments such as Embedded Swift, Explicitly Build Modules and a myriad of other improvements, there is no word, update or even a hint of what timeline we can expect here.
3. Even though Swift is OSS, and swift-syntax is OSS, there seems to be no clear way for folks to pitch in here. I know many folks (myself included) who would love to contribute to solving the problem in both discussion and actual PRs, but I don't see any roadmap or checklist to inform a path towards the solution.
4. This would be a completely different story of-course, if one could truly fork SwiftPM and plug their fork into Xcode, but Xcode uses a private fork of SwiftPM and it is not viable for a majority of Swift developers to switch to a custom fork of SwiftPM interim even if they're motivated enough to temporarily patch the issue.

It's extremely demoralizing to be engaging with this as "an issue of an OSS project". My two cents, if this is an issue which the OSS community cannot meaningfully contribute to and is linked/blocked for whatever (I'm not here to speculate) internal thing, we should perhaps consider making that clear by way of maybe a label and an explicit disclaimer. I'd love to know if there is already some existing document/disclosure of policy that addresses this.

I'm only saying this because to the best of my knowledge, the LSG's meeting notes nor the OSS roadmap discussion have touched upon what exactly is going on here (please do correct me if I'm wrong!).

[lyzkov]

@vmanot

There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).

In my opinion, Swift Package Manager should possess the ability to patch the whole dependency graph with a repository fork that has a compatible Package.swift manifest. There are already a lot of package managers (Cargo) in other languages that utilize such overriding possibility for developers who don't want to fork all sub-dependent packages from the dependency graph. For security and readability reasons, it would be necessary to introduce an explicit patch delimiter, i.e., Package.Dependency.patch(url:from:). As such, a top-level, most dependent package will have the ability to alter any of its dependencies. Is anyone in contact with SPM devs?

[ahoppen]

SwiftPM should support this already using dependency mirrors: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md