[Performance]: Performance comparison for v1 engine and v0 engine

[hustxiayang]

Proposal to improve performance

No response

Report of performance regression

Hi, I did a benchmark to compare the performance of v1 engine and v0 engine, using benchmark_serving.py on SharGPT and an internal dataset.

The results for llama3-3-70b-instruct are shown as follows.
ShareGPT:

Our internal dataset:

The average length prompts in our internal dataset is around 9k tokens.
It seems that the performance of the v1 engine is much worse, and it seems that TTFT is much larger for long prompts under a high QPS v1 engine.

For setup details: I used 4 H100s and kserve for both deployments. I used vLLM-0.8.4, using quantization fp8-dynamic.
The launch params for v0 engine:

       '--gpu-memory-utilization=0.90'
       '--tensor-parallel-size=4'
       '--enable-chunked-prefill'
       '--max-num-batched-tokens=8192'
       --enable-auto-tool-choice
       --tool-call-parser=llama3_json

The launch params for v1 engine:

      '--gpu-memory-utilization=0.90'
       '--tensor-parallel-size=4'
       --enable-auto-tool-choice
       --tool-call-parser=llama3_json

The only difference is that for v1 engine, it does not need enable-chunked-prefill and max-num-batched-tokens is already 8192.

Let me know whether it’s a fair comparison.

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xsank]

Mark, i have the same confusion and still work on it.

[zkf-qwj]

@hustxiayang
Hi, Could you supplement the hardware environment and vllm version information?

[hustxiayang]

@zkf-qwj it's 4 H100 with tp=4, vLLM version=0.8.4

[Zerohertz]

@hustxiayang

Thank you for this detailed performance comparison between the v0 and v1 engines and for sharing your benchmark results, especially highlighting the TTFT differences observed with your internal dataset on long prompts.

To help the vLLM team and the community better understand, reproduce, and investigate the performance characteristics you've reported, would it be possible for you to share the internal dataset used for these benchmarks? If so, a link to it (e.g., on Hugging Face Hub or another accessible platform) would be greatly appreciated.

If sharing the dataset directly is not feasible due to its proprietary nature or other constraints, could you perhaps provide some more detailed characteristics of the data? For instance, beyond the average prompt length of ~9k tokens, information like:

• The distribution of prompt lengths (e.g., min, max, median, percentiles).
• The general nature or domain of the prompts (e.g., are they QA, summarization, conversational, code generation, specific instruction-following tasks?).
• Any other structural properties or common patterns within the prompts that might be relevant to how the vLLM engine processes them (e.g., presence of many few-shot examples, specific formatting, etc.).

This additional information could be very valuable for diagnosing the observed performance differences with the v1 engine on long prompts.

Thanks again for your thorough testing and for raising this important issue!

[Zerohertz]

Regarding the issue you shared, I conducted tests using the same command on a system with 4×H100 GPUs.

# V0
$ VLLM_USE_V1=0 python -m vllm.entrypoints.openai.api_server \
	--model meta-llama/Llama-3.3-70B-Instruct \
	--gpu-memory-utilization=0.90 \
	--tensor-parallel 4 \
	--enable-chunked-prefill \
	--max-num-batched-tokens=8192 \
	--enable-auto-tool-choice \
	--tool-call-parser=llama3_json

# V1
$ VLLM_USE_V1=1 python -m vllm.entrypoints.openai.api_server \
	--model meta-llama/Llama-3.3-70B-Instruct \
	--gpu-memory-utilization=0.90 \
	--tensor-parallel 4 \
	--enable-auto-tool-choice \
	--tool-call-parser=llama3_json

$ python benchmarks/benchmark_serving.py \
	--backend vllm \
	--model meta-llama/Llama-3.3-70B-Instruct \
	--dataset-name sharegpt \
	--dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
	--seed 42 \
	--num-prompts 100 \
	--request-rate ${RPS} \
	--burstiness 1.0 \
	--percentile-metrics "ttft,tpot,itl,e2el" \
	--save-result

The results are as follows:

When comparing the TTFT values, there are noticeable differences from my results.
Additionally, the trend in E2E latency also differs.
In your experiment, there was a significant gap in E2E latency between the v0 and v1 engines as QPS increased, whereas in my environment, the difference was minimal.

[Zerohertz]

Therefore, to simulate long input (9K tokens), we conducted the benchmark again using random input as shown below:

$ python benchmarks/benchmark_serving.py \
	--backend vllm \
	--model meta-llama/Llama-3.3-70B-Instruct \
	--dataset-name random \
	--random-input-len 9000 \
	--random-output-len 128 \
	--seed 42 \
	--num-prompts 100 \
	--request-rate ${RPS} \
	--burstiness 1.0 \
	--percentile-metrics "ttft,tpot,itl,e2el" \
	--save-result

The results are as follows:

As can be seen from the results, both E2E latency and TTFT improved in v1 compared to v0 across various QPS levels.
However, since this experiment was conducted using random input data—not the same data used by @hustxiayang, nor realistic data—we cannot definitively conclude that there are no issues with the v1 engine.

[hustxiayang]

@Zerohertz Thanks a lot for your detailed performance comparison! Did you also use quantized version: fp8-dynamic from NM/RedHat? Thanks a lot!

[Zerohertz]

@hustxiayang

I've tried RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic too.

MODEL=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic

# V0
VLLM_USE_V1=0 python -m vllm.entrypoints.openai.api_server \
	--model ${MODEL} \
	--gpu-memory-utilization 0.90 \
	--tensor-parallel 4 \
	--enable-chunked-prefill \
	--max-num-batched-tokens 8192 \
	--enable-auto-tool-choice \
	--tool-call-parser llama3_json \
	--disable-log-requests

# V1
VLLM_USE_V1=1 python -m vllm.entrypoints.openai.api_server \
	--model ${MODEL} \
	--gpu-memory-utilization 0.90 \
	--tensor-parallel 4 \
	--enable-auto-tool-choice \
	--tool-call-parser llama3_json \
	--disable-log-requests

[tjtanaa]

Why the v0 and v1 engine has the same performance and v0 engine can be slightly faster than v1 engine?

2 * A100
vLLM v0.9.0

$ python3 -m pip list | grep cuda
cupy-cuda12x                             13.4.1
nvidia-cuda-cupti-cu12                   12.6.80
nvidia-cuda-nvrtc-cu12                   12.6.77
nvidia-cuda-runtime-cu12                 12.6.77

v0
============ Serving Benchmark Result ============
Successful requests:                     500       
Benchmark duration (s):                  1277.61   
Total input tokens:                      499500    
Total generated tokens:                  221603    
Request throughput (req/s):              0.39      
Request goodput (req/s):                 0.00      
Output token throughput (tok/s):         173.45    
Total Token throughput (tok/s):          564.41    
---------------Time to First Token----------------
Mean TTFT (ms):                          589195.33 
Median TTFT (ms):                        605856.06 
P99 TTFT (ms):                           1214914.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          114.70    
Median TPOT (ms):                        83.17     
P99 TPOT (ms):                           939.46    
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.77     
Median ITL (ms):                         61.44     
P99 ITL (ms):                            961.58    
==================================================

v1
============ Serving Benchmark Result ============
Successful requests:                     500       
Benchmark duration (s):                  1301.77   
Total input tokens:                      499500    
Total generated tokens:                  219449    
Request throughput (req/s):              0.38      
Request goodput (req/s):                 0.00      
Output token throughput (tok/s):         168.58    
Total Token throughput (tok/s):          552.29    
---------------Time to First Token----------------
Mean TTFT (ms):                          610385.56 
Median TTFT (ms):                        594344.58 
P99 TTFT (ms):                           1241401.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          144.00    
Median TPOT (ms):                        80.05     
P99 TPOT (ms):                           1537.61   
---------------Inter-token Latency----------------
Mean ITL (ms):                           83.85     
Median ITL (ms):                         61.37     
P99 ITL (ms):                            575.79    
==================================================

v0 server command
VLLM_USE_V1=0 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct -tp 2 --max-num-seqs 128 --max-model-len 10000

v1 server command
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct -tp 2 --max-num-seqs 128 --max-model-len 10000

client:

python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Meta-Llama-3.1-70B-Instruct \
    --dataset-name random \
    --num-prompts 500 \
    --goodput "ttft:3000" "tpot:50" \
    --random-input-len 1000 \
    --port 8007 \
    --random-output-len 1000