[infra] directly expose passing logs for flaky tests

[sigmundch]

When investigating flaky failures, I found it valuable to access both the failure logs and the passing logs. For example:

• When there is a flaky timeout, it helps to see how long test takes to run on a passing configuration, for comparison. Tests that are bimodal need to be fixed differently than tests that are always running near the limit.
• When there is a flaky failure, it is often helpful to add diagnostic messages to the tests to narrow down the potential cause and compare it against a successful run.

Today we easily find the logs for a recent failure of a flaky test:

• links are provided in the flakiness dashboard for the most recent failure
• on any builder, there is a ignored_flaky_test_failure_logs link with logs for what failed during that run.

Would it be possible to make similar data available for flaky passes? For example, we could have a ignored_flaky_test_passing_logs? Or a link in the dashboard for the last "passing" column?

[whesse]

At the deepest levels of the test runner, logs are not even captured for passing tests. Whether a test is flaky or not is not something that is passed down into the test runner process, so the test runner would currently need to record and output the logs of all tests. Timing information is available for the passing tests, in the results.json records. I have made bigquery queries, and an app that downoads timing data from recent runs, to investigate timing discrepancies, and whether the timeouts are bimodal or not, so that doesn't really connect with the logs issue.

Would passing tests produce useful output in the logs? And if so, would it also be produced by a local passing run? We have usually had the idea that passing tests will produce no useful output. So are there really things to get from a passing run on the CI that aren't gotten by seeing the timing information and the log of a local passing run?

A lot of refactoring and implementation would need to be done to get the stdout of a passing run. I don't think we can manage to pass the --verbose flag in, but that might do it.