[Benchmark][V1][Spec Decode][EAGLE] Tracking benchmark for V1 EAGLE

[ekagra-ranjan]

We have been doing perf bench on MTBench so that e2e speedup and AL are comparable with other setups and academic papers. Thanks to @luyuzhe111 and others for the discussion and helping with measuring the gaps!

llama 3 8b

During model wt loading

• [V1][Spec Decode] Eagle Model loading #16035 (comment)
  • SGL correction: [V1][Spec Decode] Eagle Model loading #16035 (comment)
  • SGL setup: https://docs.google.com/document/d/18ETJLsnxR88Qq3VDk5Mq-Hb7vuE9o3VNZ-hhz-OqAXk/edit?usp=sharing
• [V1][Spec Decode] Eagle Model loading #16035 (comment)

During KV Cache slot

• [V1][Spec Decode] KV cache slots for eagle heads #16370 (comment)

llama 3.1 8b

• [V1][Spec Decode] KV cache slots for eagle heads #16370 (comment)
• EAGLE - 1/3
  • offline serving: [V1][Spec Decode] EAGLE-3 Support #16937 (comment)
  • online serving: [Benchmark] Add single turn MTBench to Serving Bench #17202 (comment)

torch compile & CUDA graph:

• EAGLE 1
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE #17211 (comment)
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE #17211 (comment)
• EAGLE 3
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE3 #17504 (comment)

[wwl2755]

Great job and thanks on collecting these!

Do you have a unified table/doc that keeps all the up-to-date benchmarking (and also the gaps with ideal condition), which I believe would be much helpful?

[wwl2755]

And also, making the metrics-related PRs (#16367 and #17010) finialized and merged would be great.

[ekagra-ranjan]

Do you have a unified table/doc that keeps all the up-to-date benchmarking (and also the gaps with ideal condition), which I believe would be much helpful?

I didnt get the time to do it. I believe the most recent comment would be the one to refer.