Skip to content

Remove Benchmark stages #6895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 8, 2025
Merged

Remove Benchmark stages #6895

merged 3 commits into from
May 8, 2025

Conversation

andrewlock
Copy link
Member

@andrewlock andrewlock commented Apr 25, 2025 •
edited
Loading

Summary of changes

Removes the Benchmark stages that currently run in Azure

Reason for change

We now run the Benchmarks in the microbenchmarking platform, so these are obsolete.

Implementation details

Delete the benchmarks and appsec_benchmarks stages. Note that the benchmarking platform tests (defined here: https://github.com/DataDog/benchmarking-platform/tree/dd-trace-dotnet/micro) still run the same RunBenchmarks Nuke target to run the tests

Test coverage

Essentially the same

Other details

Fixed one weird issue where we were running into null refs comparing the AppSec benchmarks where they weren't always available. Will dig into that further later, but I think this is OK for now

We should only merge this once the following is merged, and we're correctly uploading our benchmark results:

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 25, 2025
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Apr 25, 2025 •
edited
Loading

Datadog Report

Branch report: andrew/ci/remove-benchmarks
Commit report: 9a7a403
Test service: dd-trace-dotnet

✅ 0 Failed, 249865 Passed, 2260 Skipped, 17h 33m 6.73s Total Time

@andrewlock
Copy link
Member Author

andrewlock commented Apr 25, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (1,003ms)  : 982, 1023
     .   : milestone, 1003,
    master - mean (1,009ms)  : 985, 1034
     .   : milestone, 1009,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (103ms)  : 100, 105
     .   : milestone, 103,
    master - mean (103ms)  : 101, 105
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (694ms)  : 679, 709
     .   : milestone, 694,
    master - mean (697ms)  : 681, 714
     .   : milestone, 697,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (90ms)  : 88, 92
     .   : milestone, 90,
    master - mean (90ms)  : 88, 91
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (654ms)  : 635, 673
     .   : milestone, 654,
    master - mean (656ms)  : 635, 677
     .   : milestone, 656,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (189ms)  : 185, 194
     .   : milestone, 189,
    master - mean (191ms)  : 184, 197
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (1,105ms)  : 1080, 1131
     .   : milestone, 1105,
    master - mean (1,110ms)  : 1084, 1136
     .   : milestone, 1110,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (269ms)  : 266, 272
     .   : milestone, 269,
    master - mean (271ms)  : 267, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (881ms)  : 846, 916
     .   : milestone, 881,
    master - mean (881ms)  : 848, 914
     .   : milestone, 881,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6895) - mean (262ms)  : 257, 266
     .   : milestone, 262,
    master - mean (261ms)  : 257, 265
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6895) - mean (865ms)  : 837, 893
     .   : milestone, 865,
    master - mean (869ms)  : 837, 901
     .   : milestone, 869,
Loading

@andrewlock andrewlock force-pushed the andrew/ci/remove-benchmarks branch from 10036fb to 9a7a403 Compare May 8, 2025 10:16
@andrewlock andrewlock added the area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc label May 8, 2025
@pr-commenter
Copy link

pr-commenter bot commented May 8, 2025

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #6895 compared to master:

  • All benchmarks have the same speed
  • 3 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6895

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.21 KB 6.15 KB -53 B -0.85%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.6μs 60ns 416ns 0 0 0 5.56 KB
master StartStopWithChild netcoreapp3.1 14.2μs 73.2ns 366ns 0 0 0 5.72 KB
master StartStopWithChild net472 22.7μs 124ns 756ns 0.904 0.226 0 6.21 KB
#6895 StartStopWithChild net6.0 10.5μs 58.1ns 348ns 0 0 0 5.55 KB
#6895 StartStopWithChild netcoreapp3.1 13.5μs 63.2ns 245ns 0 0 0 5.72 KB
#6895 StartStopWithChild net472 21.9μs 119ns 662ns 0.894 0.224 0 6.15 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 937μs 55.1ns 199ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 78.5ns 283ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.23ms 850ns 3.29μs 0 0 0 3.31 KB
#6895 WriteAndFlushEnrichedTraces net6.0 924μs 113ns 422ns 0 0 0 2.7 KB
#6895 WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 80.7ns 291ns 0 0 0 2.7 KB
#6895 WriteAndFlushEnrichedTraces net472 1.23ms 46.2ns 167ns 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61μs 96.2ns 347ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 70.5μs 157ns 544ns 0 0 0 17.42 KB
master SendRequest net472 0.0178ns 0.00489ns 0.0189ns 0 0 0 0 b
#6895 SendRequest net6.0 62μs 79.7ns 309ns 0 0 0 14.53 KB
#6895 SendRequest netcoreapp3.1 70.8μs 46ns 166ns 0 0 0 17.42 KB
#6895 SendRequest net472 2.25E‑05ns 2.25E‑05ns 8.7E‑05ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6895

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 56.36 KB 55.83 KB -529 B -0.94%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 42.54 KB 41.95 KB -585 B -1.38%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 706μs 1.16μs 4.5μs 0 0 0 41.59 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 769μs 4.37μs 31.2μs 0 0 0 42.54 KB
master WriteAndFlushEnrichedTraces net472 868μs 2.47μs 8.92μs 8.33 0 0 56.36 KB
#6895 WriteAndFlushEnrichedTraces net6.0 661μs 1.37μs 5.29μs 0 0 0 41.69 KB
#6895 WriteAndFlushEnrichedTraces netcoreapp3.1 739μs 4.3μs 37.8μs 0 0 0 41.95 KB
#6895 WriteAndFlushEnrichedTraces net472 924μs 2.46μs 9.54μs 4.46 0 0 55.83 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 2.09μs 8.62ns 33.4ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.53μs 8.98ns 34.8ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.78μs 2.45ns 9.49ns 0.153 0.0139 0 995 B
#6895 ExecuteNonQuery net6.0 1.9μs 2.06ns 7.97ns 0 0 0 1.03 KB
#6895 ExecuteNonQuery netcoreapp3.1 2.42μs 12.2ns 54.4ns 0 0 0 1.02 KB
#6895 ExecuteNonQuery net472 2.79μs 6.33ns 24.5ns 0.152 0.0138 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.8μs 6.35ns 23.8ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.26μs 10.5ns 40.5ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.56μs 2.3ns 8.29ns 0.161 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.82μs 4.48ns 16.8ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.31μs 10.1ns 39.1ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.59μs 1.84ns 7.13ns 0.162 0 0 1.11 KB
#6895 CallElasticsearch net6.0 1.74μs 9.13ns 43.8ns 0 0 0 1.04 KB
#6895 CallElasticsearch netcoreapp3.1 2.31μs 8.77ns 34ns 0 0 0 1.04 KB
#6895 CallElasticsearch net472 3.52μs 1.12ns 4.32ns 0.158 0 0 1.05 KB
#6895 CallElasticsearchAsync net6.0 1.83μs 8.16ns 31.6ns 0 0 0 1.02 KB
#6895 CallElasticsearchAsync netcoreapp3.1 2.51μs 3.22ns 12.5ns 0 0 0 1.09 KB
#6895 CallElasticsearchAsync net472 3.76μs 4.27ns 16.5ns 0.17 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.9μs 6.95ns 26.9ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.39μs 11ns 42.6ns 0 0 0 960 B
master ExecuteAsync net472 2.58μs 7.02ns 27.2ns 0.144 0 0 923 B
#6895 ExecuteAsync net6.0 1.94μs 6.1ns 23.6ns 0 0 0 960 B
#6895 ExecuteAsync netcoreapp3.1 2.42μs 9.66ns 37.4ns 0 0 0 960 B
#6895 ExecuteAsync net472 2.63μs 3.87ns 15ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.88μs 9.66ns 37.4ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.43μs 36.2ns 140ns 0 0 0 2.9 KB
master SendAsync net472 12.2μs 4.02ns 14.5ns 0.488 0 0 3.19 KB
#6895 SendAsync net6.0 7.14μs 14.3ns 53.4ns 0 0 0 2.37 KB
#6895 SendAsync netcoreapp3.1 8.59μs 12.4ns 47.9ns 0 0 0 2.9 KB
#6895 SendAsync net472 11.9μs 13.4ns 51.9ns 0.473 0 0 3.19 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.64μs 13ns 51.9ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.46μs 18.1ns 87ns 0 0 0 1.76 KB
master EnrichedLog net472 3.98μs 3.47ns 13.4ns 0.259 0 0 1.69 KB
#6895 EnrichedLog net6.0 2.53μs 11.8ns 47.2ns 0 0 0 1.76 KB
#6895 EnrichedLog netcoreapp3.1 3.62μs 14.8ns 57.4ns 0 0 0 1.76 KB
#6895 EnrichedLog net472 3.9μs 4.54ns 16.4ns 0.255 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 122μs 45.5ns 164ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 131μs 620ns 2.4μs 0 0 0 4.37 KB
master EnrichedLog net472 168μs 340ns 1.32μs 0 0 0 4.57 KB
#6895 EnrichedLog net6.0 124μs 114ns 409ns 0 0 0 4.37 KB
#6895 EnrichedLog netcoreapp3.1 129μs 286ns 1.07μs 0 0 0 4.37 KB
#6895 EnrichedLog net472 168μs 89.7ns 336ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.16μs 4.51ns 16.9ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.58μs 14.9ns 57.6ns 0 0 0 2.32 KB
master EnrichedLog net472 7.36μs 7.51ns 28.1ns 0.33 0 0 2.14 KB
#6895 EnrichedLog net6.0 5.05μs 3.52ns 13.2ns 0 0 0 2.32 KB
#6895 EnrichedLog netcoreapp3.1 6.69μs 24ns 89.7ns 0 0 0 2.32 KB
#6895 EnrichedLog net472 7.4μs 6.14ns 23.8ns 0.334 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.04μs 0.788ns 2.84ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.61μs 10.4ns 40.4ns 0 0 0 1.21 KB
master SendReceive net472 3.14μs 1.67ns 6.45ns 0.188 0 0 1.21 KB
#6895 SendReceive net6.0 1.98μs 9.17ns 36.7ns 0 0 0 1.21 KB
#6895 SendReceive netcoreapp3.1 2.65μs 11ns 42.8ns 0 0 0 1.21 KB
#6895 SendReceive net472 3.08μs 2.82ns 10.5ns 0.187 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.17μs 13.2ns 49.2ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.77μs 12ns 46.6ns 0 0 0 1.69 KB
master EnrichedLog net472 6.62μs 8.54ns 33.1ns 0.329 0 0 2.08 KB
#6895 EnrichedLog net6.0 4.22μs 15.5ns 60.1ns 0 0 0 1.64 KB
#6895 EnrichedLog netcoreapp3.1 5.55μs 17.3ns 67.1ns 0 0 0 1.69 KB
#6895 EnrichedLog net472 6.59μs 10.8ns 41.9ns 0.328 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 758ns 3.92ns 18ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 947ns 4.74ns 22.2ns 0 0 0 584 B
master StartFinishSpan net472 923ns 0.0955ns 0.37ns 0.0924 0 0 586 B
master StartFinishScope net6.0 898ns 1.72ns 6.66ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.16μs 5.63ns 22.5ns 0 0 0 704 B
master StartFinishScope net472 1.1μs 0.266ns 0.995ns 0.106 0 0 666 B
#6895 StartFinishSpan net6.0 754ns 3.51ns 14.5ns 0 0 0 584 B
#6895 StartFinishSpan netcoreapp3.1 952ns 5.12ns 26.6ns 0 0 0 584 B
#6895 StartFinishSpan net472 888ns 0.249ns 0.965ns 0.0888 0 0 586 B
#6895 StartFinishScope net6.0 900ns 4.74ns 24.6ns 0 0 0 704 B
#6895 StartFinishScope netcoreapp3.1 1.18μs 2.17ns 8.39ns 0 0 0 704 B
#6895 StartFinishScope net472 1.08μs 0.187ns 0.701ns 0.102 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.04μs 5.06ns 20.9ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.41μs 6.7ns 26ns 0 0 0 704 B
master RunOnMethodBegin net472 1.41μs 0.718ns 2.78ns 0.104 0 0 666 B
#6895 RunOnMethodBegin net6.0 1.02μs 2.84ns 11ns 0 0 0 704 B
#6895 RunOnMethodBegin netcoreapp3.1 1.4μs 5.81ns 22.5ns 0 0 0 704 B
#6895 RunOnMethodBegin net472 1.34μs 0.736ns 2.55ns 0.101 0 0 666 B

@andrewlock andrewlock marked this pull request as ready for review May 8, 2025 11:47
@andrewlock andrewlock requested a review from a team as a code owner May 8, 2025 11:47
bouwkast
bouwkast approved these changes May 8, 2025
View reviewed changes
Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@andrewlock andrewlock merged commit 30b5231 into master May 8, 2025
127 checks passed
@andrewlock andrewlock deleted the andrew/ci/remove-benchmarks branch May 8, 2025 12:53
@github-actions github-actions bot added this to the vNext-v3 milestone May 8, 2025
@andrewlock andrewlock mentioned this pull request May 8, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bouwkast bouwkast bouwkast approved these changes

@ganeshnj ganeshnj ganeshnj approved these changes

Assignees
No one assigned
Labels
area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.17.0
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewlock @bouwkast @ganeshnj