Benchmarking work checklist

[bf4]

Now that #1393 (comment) is merged, there's some follow-up work to do and consider:

• Docs
  • when and how devs should use
  • command line options to bin/bench
bin/bench --help

Usage: bin/bench [options]

-r, --repeat-count [NUM] Run benchmarks [NUM] times taking the best result

-p <PATTERN1,PATTERN2,PATTERN3>, Benchmark name pattern

--pattern

-e <var1=val1,var2=val2,var3=vale>,

--env ENV variables to pass in
  • ENV vars that can be passed to bin/bench
    • SUMMARIZE, CACHE_ON, FAIL_ASSERTION, DEBUG, BENCH_STRESS, ENABLE_ACTIVE_RECORD
  • adding another runner (just model off of exiting)
• consider: consolidate dummy app with isolated app and test helper app
• consider renaming dummy to benchmark Rename dummy to benchmark #1576
• consider removing:
  • active record
  • bench stress
  • bin/serve_dummy
  • summary
  • logger options
• review for dead code, options, comments, etc.
• add Rails version to output Setup benchmark testing #1393 (comment)
• after merge followup with @ianks in Add benchmarks for adapters #1511
• decide what goes in a followup pr
  • running in CI?
  • running in ruby-bench-suite or our own?
  • running against actioncontroller::base and actioncontroller::API
  • regression runner (across revisions)
  • FakeIPS for running without needing to vendor benchmark-ips
  • derailed benchmarks gemfile
  • rack-mini-profiler