Skip to content

Don't merge sample build artifacts #6851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 15, 2025
Merged

Conversation

andrewlock
Copy link
Member

Summary of changes

Don't bother to "merge" the samples into a single artifact to reduce download size

Reason for change

When we moved the sample build to a separate stage (to avoid re-work and to run the build in parallel) we added a job that would aggregate all the different TFMs into a single artifact. This was for simplicity on the consuming side.

However, the size of this merged sample artifact was huge, such that it takes ~6 minutes to download again in some cases (the arm64 agents are much slower to do the download). In general, only a single TFM is ever used anyway (because we parallelize our tests by TFM too), so this is mostly unnecessarily wasted time.

Implementation details

  • Remove the merge job in the build_samples stage
  • Add a "helper" download-samples.yml script to make sure you download the different samples to the correct folder (standalone goes in bin, multi-version goes in publish currently)
  • Update usages

Test coverage

  • Ran an "all areas" test here
  • Ran an "all TFMs" test here

Other details

Removing the merge job also speeds up the build_samples stage which is sometimes a blocker for testing stages, so that's an extra bonus

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 14, 2025
@andrewlock andrewlock requested a review from a team as a code owner April 14, 2025 13:22
@andrewlock andrewlock changed the title Split sample upload Don't merge sample build artifacts Apr 14, 2025
artifact: samples
path: $(outputDir)
retryCountOnTaskFailure: 5
- template: steps/download-samples.yml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't currently run multi-version tests on windows, so don't need to specify the framework here

@andrewlock
Copy link
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (70ms)  : 66, 74
     .   : milestone, 70,
    master - mean (70ms)  : 66, 73
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (1,014ms)  : 993, 1034
     .   : milestone, 1014,
    master - mean (1,013ms)  : 984, 1041
     .   : milestone, 1013,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (104ms)  : 102, 106
     .   : milestone, 104,
    master - mean (103ms)  : 101, 105
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (702ms)  : 685, 718
     .   : milestone, 702,
    master - mean (695ms)  : 675, 714
     .   : milestone, 695,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (91ms)  : 88, 93
     .   : milestone, 91,
    master - mean (90ms)  : 88, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (656ms)  : 632, 679
     .   : milestone, 656,
    master - mean (656ms)  : 633, 678
     .   : milestone, 656,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (190ms)  : 185, 196
     .   : milestone, 190,
    master - mean (189ms)  : 185, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (1,110ms)  : 1080, 1140
     .   : milestone, 1110,
    master - mean (1,105ms)  : 1084, 1126
     .   : milestone, 1105,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (268ms)  : 265, 272
     .   : milestone, 268,
    master - mean (268ms)  : 264, 273
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (879ms)  : 847, 910
     .   : milestone, 879,
    master - mean (881ms)  : 855, 908
     .   : milestone, 881,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6851) - mean (261ms)  : 257, 266
     .   : milestone, 261,
    master - mean (261ms)  : 258, 264
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6851) - mean (870ms)  : 832, 907
     .   : milestone, 870,
    master - mean (867ms)  : 829, 905
     .   : milestone, 867,

Loading

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Apr 14, 2025

Datadog Report

All test runs c0802f0 🔗

2 Total Test Services: 0 Failed, 2 Passed
❄️ 1 with New Flaky

Test Services
Service Name Failed Known Flaky New Flaky Passed Skipped Total Time Test Service View
dd-trace-dotnet 0 0 4 590146 6227 91h 24m 47.11s Link
exploration_tests 0 0 0 22085 3 14m 45.33s Link

@andrewlock
Copy link
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #6851 compared to master:

  • 3 benchmarks are faster, with geometric mean 1.171
  • 2 benchmarks are slower, with geometric mean 1.162
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8μs 45.4ns 337ns 0.0153 0.00767 0 5.56 KB
master StartStopWithChild netcoreapp3.1 10.4μs 55.6ns 289ns 0.0248 0.00994 0 5.75 KB
master StartStopWithChild net472 15.8μs 37.3ns 140ns 1.02 0.297 0.102 6.15 KB
#6851 StartStopWithChild net6.0 7.96μs 85.7ns 853ns 0 0 0 5.57 KB
#6851 StartStopWithChild netcoreapp3.1 10.1μs 126ns 1.26μs 0 0 0 5.74 KB
#6851 StartStopWithChild net472 15.6μs 73.1ns 301ns 1.05 0.3 0.075 6.14 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 481μs 165ns 619ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 658μs 267ns 999ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 857μs 774ns 3μs 0.428 0 0 3.3 KB
#6851 WriteAndFlushEnrichedTraces net6.0 495μs 397ns 1.49μs 0 0 0 2.7 KB
#6851 WriteAndFlushEnrichedTraces netcoreapp3.1 674μs 782ns 2.82μs 0 0 0 2.7 KB
#6851 WriteAndFlushEnrichedTraces net472 862μs 575ns 2.23μs 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 130μs 383ns 1.48μs 0.139 0 0 14.48 KB
master SendRequest netcoreapp3.1 147μs 336ns 1.3μs 0.222 0 0 17.28 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#6851 SendRequest net6.0 130μs 721ns 4.62μs 0 0 0 14.48 KB
#6851 SendRequest netcoreapp3.1 150μs 479ns 1.85μs 0 0 0 17.28 KB
#6851 SendRequest net472 0.00115ns 0.000577ns 0.00216ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 579μs 2.97μs 14.2μs 0.556 0 0 41.65 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 678μs 3.81μs 25μs 0.345 0 0 41.91 KB
master WriteAndFlushEnrichedTraces net472 869μs 4.01μs 15.5μs 8.25 2.6 0.434 53.3 KB
#6851 WriteAndFlushEnrichedTraces net6.0 548μs 2.08μs 8.05μs 0 0 0 41.57 KB
#6851 WriteAndFlushEnrichedTraces netcoreapp3.1 666μs 3.17μs 15.2μs 0 0 0 41.82 KB
#6851 WriteAndFlushEnrichedTraces net472 848μs 4.93μs 44.3μs 7.81 3.91 0 53.47 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.36μs 0.966ns 3.74ns 0.0143 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 1.82μs 1.39ns 5.4ns 0.0136 0 0 1.02 KB
master ExecuteNonQuery net472 2.04μs 2.21ns 8.54ns 0.158 0.00102 0 995 B
#6851 ExecuteNonQuery net6.0 1.37μs 1.36ns 5.1ns 0.0134 0 0 1.03 KB
#6851 ExecuteNonQuery netcoreapp3.1 1.83μs 1.64ns 5.9ns 0.00915 0 0 1.02 KB
#6851 ExecuteNonQuery net472 2.04μs 3.74ns 14ns 0.149 0.00993 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6851

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.143 1,285.93 1,125.07

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.29μs 0.478ns 1.66ns 0.0142 0 0 984 B
master CallElasticsearch netcoreapp3.1 1.62μs 0.919ns 3.44ns 0.0131 0 0 984 B
master CallElasticsearch net472 2.57μs 1.04ns 3.89ns 0.159 0 0 1 KB
master CallElasticsearchAsync net6.0 1.27μs 0.695ns 2.69ns 0.0134 0 0 960 B
master CallElasticsearchAsync netcoreapp3.1 1.58μs 0.661ns 2.47ns 0.0135 0 0 1.03 KB
master CallElasticsearchAsync net472 2.63μs 1.58ns 6.11ns 0.168 0 0 1.06 KB
#6851 CallElasticsearch net6.0 1.13μs 0.977ns 3.66ns 0.0113 0 0 984 B
#6851 CallElasticsearch netcoreapp3.1 1.6μs 1.22ns 4.73ns 0.00806 0 0 984 B
#6851 CallElasticsearch net472 2.52μs 1.49ns 5.56ns 0.151 0 0 1 KB
#6851 CallElasticsearchAsync net6.0 1.35μs 1.14ns 4.43ns 0.0134 0 0 960 B
#6851 CallElasticsearchAsync netcoreapp3.1 1.59μs 2ns 7.21ns 0.00804 0 0 1.03 KB
#6851 CallElasticsearchAsync net472 2.74μs 2.41ns 9.01ns 0.166 0 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.27μs 0.623ns 2.33ns 0.0135 0 0 960 B
master ExecuteAsync netcoreapp3.1 1.68μs 0.782ns 2.82ns 0.0125 0 0 960 B
master ExecuteAsync net472 1.89μs 0.715ns 2.77ns 0.147 0 0 923 B
#6851 ExecuteAsync net6.0 1.31μs 0.854ns 3.31ns 0.0133 0 0 960 B
#6851 ExecuteAsync netcoreapp3.1 1.72μs 5.91ns 22.9ns 0.00848 0 0 960 B
#6851 ExecuteAsync net472 1.84μs 0.576ns 2.23ns 0.138 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.43μs 0.878ns 3.16ns 0.0325 0 0 2.32 KB
master SendAsync netcoreapp3.1 5.35μs 2.14ns 8.03ns 0.0373 0 0 2.86 KB
master SendAsync net472 7.43μs 2.06ns 7.72ns 0.495 0.00372 0 3.13 KB
#6851 SendAsync net6.0 4.42μs 3.1ns 12ns 0.0224 0 0 2.32 KB
#6851 SendAsync netcoreapp3.1 5.34μs 8.17ns 31.6ns 0.0266 0 0 2.86 KB
#6851 SendAsync net472 7.47μs 4.61ns 17.9ns 0.485 0 0 3.13 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.52μs 1.07ns 4.15ns 0.0241 0 0 1.71 KB
master EnrichedLog netcoreapp3.1 2.13μs 0.724ns 2.71ns 0.0228 0 0 1.71 KB
master EnrichedLog net472 2.66μs 1.25ns 4.85ns 0.26 0 0 1.64 KB
#6851 EnrichedLog net6.0 1.55μs 2.28ns 8.53ns 0.0232 0 0 1.71 KB
#6851 EnrichedLog netcoreapp3.1 2.34μs 4.82ns 18ns 0.0229 0 0 1.71 KB
#6851 EnrichedLog net472 2.72μs 1.68ns 6.51ns 0.258 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 141ns 546ns 0.0564 0 0 4.32 KB
master EnrichedLog netcoreapp3.1 116μs 245ns 951ns 0.0575 0 0 4.32 KB
master EnrichedLog net472 150μs 176ns 682ns 0.673 0.224 0 4.51 KB
#6851 EnrichedLog net6.0 111μs 148ns 555ns 0 0 0 4.32 KB
#6851 EnrichedLog netcoreapp3.1 118μs 309ns 1.16μs 0 0 0 4.32 KB
#6851 EnrichedLog net472 152μs 367ns 1.42μs 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3μs 1.07ns 4.13ns 0.0315 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 4.17μs 1.67ns 6.25ns 0.0292 0 0 2.26 KB
master EnrichedLog net472 5.05μs 1.73ns 6.69ns 0.331 0 0 2.09 KB
#6851 EnrichedLog net6.0 2.86μs 2.39ns 8.93ns 0.0287 0 0 2.26 KB
#6851 EnrichedLog netcoreapp3.1 4.14μs 2.64ns 9.52ns 0.0206 0 0 2.26 KB
#6851 EnrichedLog net472 4.95μs 1.81ns 6.77ns 0.322 0 0 2.09 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.42μs 0.803ns 3.11ns 0.0162 0 0 1.15 KB
master SendReceive netcoreapp3.1 1.74μs 1.18ns 4.57ns 0.0157 0 0 1.15 KB
master SendReceive net472 2.09μs 0.841ns 3.26ns 0.184 0 0 1.16 KB
#6851 SendReceive net6.0 1.39μs 0.956ns 3.7ns 0.0138 0 0 1.15 KB
#6851 SendReceive netcoreapp3.1 1.85μs 1.75ns 6.56ns 0.00921 0 0 1.15 KB
#6851 SendReceive net472 2.13μs 1.19ns 4.6ns 0.181 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.8μs 1.23ns 4.78ns 0.0238 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 4μs 1.99ns 7.72ns 0.022 0 0 1.69 KB
master EnrichedLog net472 4.51μs 2.02ns 7.81ns 0.33 0 0 2.08 KB
#6851 EnrichedLog net6.0 2.78μs 2.04ns 7.9ns 0.0138 0 0 1.64 KB
#6851 EnrichedLog netcoreapp3.1 3.73μs 1.71ns 6.17ns 0.0188 0 0 1.69 KB
#6851 EnrichedLog net472 4.47μs 4.64ns 18ns 0.314 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6851

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.186 678.16 804.24

Faster 🎉 in #6851

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.205 671.97 557.77

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 399ns 0.141ns 0.544ns 0.00819 0 0 584 B
master StartFinishSpan netcoreapp3.1 671ns 0.347ns 1.34ns 0.00777 0 0 584 B
master StartFinishSpan net472 629ns 0.252ns 0.976ns 0.0929 0 0 586 B
master StartFinishScope net6.0 489ns 0.175ns 0.679ns 0.00998 0 0 704 B
master StartFinishScope netcoreapp3.1 678ns 0.224ns 0.869ns 0.00949 0 0 704 B
master StartFinishScope net472 907ns 0.155ns 0.579ns 0.106 0 0 666 B
#6851 StartFinishSpan net6.0 402ns 0.226ns 0.847ns 0.00823 0 0 584 B
#6851 StartFinishSpan netcoreapp3.1 558ns 0.539ns 2.09ns 0.00559 0 0 584 B
#6851 StartFinishSpan net472 648ns 0.114ns 0.41ns 0.0909 0 0 586 B
#6851 StartFinishScope net6.0 488ns 0.42ns 1.63ns 0.00969 0 0 704 B
#6851 StartFinishScope netcoreapp3.1 804ns 0.793ns 2.97ns 0.00807 0 0 704 B
#6851 StartFinishScope net472 863ns 0.309ns 1.07ns 0.104 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6851

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 1.138 898.11 1,022.49

Faster 🎉 in #6851

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.165 763.95 655.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 764ns 0.329ns 1.27ns 0.00993 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 898ns 0.456ns 1.64ns 0.00951 0 0 704 B
master RunOnMethodBegin net472 1.08μs 0.359ns 1.39ns 0.106 0 0 666 B
#6851 RunOnMethodBegin net6.0 656ns 0.426ns 1.65ns 0.00986 0 0 704 B
#6851 RunOnMethodBegin netcoreapp3.1 1.02μs 1.25ns 4.84ns 0.00507 0 0 704 B
#6851 RunOnMethodBegin net472 1.09μs 1.05ns 4.06ns 0.104 0 0 666 B

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Apr 14, 2025

Datadog Report

Branch report: andrew/split-sample-upload
Commit report: c0802f0
Test service: dd-trace-dotnet

✅ 0 Failed, 8 Passed, 0 Skipped, 0s Total Time

Copy link
Contributor

@NachoEchevarria NachoEchevarria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks!

@andrewlock andrewlock merged commit 5129dc8 into master Apr 15, 2025
133 of 136 checks passed
@andrewlock andrewlock deleted the andrew/split-sample-upload branch April 15, 2025 13:21
@github-actions github-actions bot added this to the vNext-v3 milestone Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants