Nightlies for security branches?

[cmb69]

Description

When I applied #16097 the other day, I was prepared to see many of our 8.1 nightly jobs fail (Linux due to not yet applied #16107, and macOS due to some build error). Checking the builds today revealed that Linux already fails due to a missing backport of #16011 (or some alternative solution), and the some of the Community steps are also failing (apparently, due to newer versions of the libraries/frameworks/tools, which are no longer compatible with PHP 8.1).

It is obviously hard to keep these builds green, if the run only occasionally (often not for weeks), and @iluuu1994 convinced me that nightly runs of all branches would be too much. And it also might not be the best idea to target security branches for all potentially relevant CI fixes. While something like #16144 (if targeted at PHP-8.1) might help, it may still be a lot of work to cater to all those issues.

So I wonder what to do. It seems to me that the Community builds don't make much sense anyway (since PHP-8.1 should be stable enough to not introduce unexpected breaks). And assuming the we'll merge #16148 (which I'm planning to do), we already have the ability to run the push workflows, which should catch the most serious issues. Okay, no Asan and MSan builds, and no uncommon configurations, but these should ideally be run before we merge security fixes anyway.

So I tend to suggest to exclude PHP-8.1 from the nightly runs. If not, we should at least resolve the most serious issues, so that the nightly runs are somewhat meaningful.

[iluuu1994]

I agree that running old branches in nightly is meaningless if they are all red. I would not object to trying to fix the 8.1 issues now and seeing how much work it is to keep them green (or mostly green, at least). It could make sense to run all branches on Monday, for example. Then, we would find new issues on 8.1 every week instead of every few months. As mentioned previously, the configurations for the branches are diverging, which is one of the main annoyances. I have not yet looked into shared workflows, which might be a solution. Another might be always checking out ./.github from master, so that we can manage it in one place.

Community builds don't make much sense anyway (since PHP-8.1 should be stable enough to not introduce unexpected breaks).

Note that community builds run with ASan, so they are actually quite useful. It is possible though that they start dropping PHP versions that still receive security patches, in that case it would definitely make sense dropping them.

[iluuu1994]

I brought this up in todays foundation meeting. Here's the summary:

We should strive for CI of security branches to stay green. Fixes to CI itself and tests should target security branches from now on. Small code fixes may be discussed on a case-by-case basis with RMs of the given security branch. Alternatively, the given errors/warnings may be ignored with -Wno- flags. Security branches should be run in the nightly workflow on a weekly basis.

Does this sound ok to you? I will now work on making the builds green, and then update the #general channel on Slack to inform everybody to send CI patches (including fixing of tests) to security branches from now on.

[cmb69]

That sounds great!

[nielsdos]

Is this resolved now by 562677a ? I would think so unless I'm missing something?

[cmb69]

I think so. Maybe @iluuu1994 wants to keep it open for a while to check whether everything works as expected.

[iluuu1994]

I think other builds still fail (freebsd). Unfortunately, I can't check as we're out of free credits.

[cmb69]

FreeBSD has been switched to GH (using emulation), and that works fine. However, macOS fails despite #16789, and despite that icu4c@74 is in PKG_CONFIG_PATH but not icu4c@76 (and the latter is not linked either).

[iluuu1994]

@cmb69 Any suggestions on that part? We should also backport all of the flakiness checks containing the message "Occasionally segfaults on macOS for unknown reasons", as this seems to happen very frequently. The last issue is a JIT issue in the community build for one of the Symfony packages, which was fixed for stable branches in GH-16858. I just asked the RMs whether it's ok to backport it. If not, I will skip the affected package.