[Performance]: SGLANG is 4 times faster than vLLM for Qwen/Qwen3-32B-AWQ

[thesillystudent]

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

vLLM command

VLLM_ATTENTION_BACKEND=FLASHINFER python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-32B-AWQ --port 8000 --gpu-memory-utilization 0.90 --tensor-parallel-size 4  --disable-log-requests --quantization awq_marlin -O3

SGLang command

python -m sglang.launch_server --model-path Qwen/Qwen3-32B-AWQ --port 8000 --tensor-parallel-size 4 --quantization awq_marlin --dtype auto --enable-torch-compile --attention-backend flashinfer --show-time-cost --enable-metrics

benchmarking command

vllm bench serve \
--model Qwen/Qwen3-32B-AWQ \
--num-prompts 50 \
--random-input-len 25000 \
--random-output-len 1024 \
--ignore-eos \
--request-rate inf \

vLLM results

============ Serving Benchmark Result ============
Successful requests:                     50        
Benchmark duration (s):                  7303.93   
Total input tokens:                      1250000   
Total generated tokens:                  51200     
Request throughput (req/s):              0.01      
Output token throughput (tok/s):         7.01      
Total Token throughput (tok/s):          178.15    
---------------Time to First Token----------------
Mean TTFT (ms):                          2945424.93
Median TTFT (ms):                        3210790.97
P99 TTFT (ms):                           6428720.18
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1013.04   
Median TPOT (ms):                        1013.40   
P99 TPOT (ms):                           1300.17   
---------------Inter-token Latency----------------
Mean ITL (ms):                           1013.04   
Median ITL (ms):                         901.30    
P99 ITL (ms):                            2331.21   
==================================================

SGLang results

============ Serving Benchmark Result ============
Successful requests:                     50        
Benchmark duration (s):                  1737.44   
Total input tokens:                      1250000   
Total generated tokens:                  51200     
Request throughput (req/s):              0.03      
Output token throughput (tok/s):         29.47     
Total Token throughput (tok/s):          748.92    
---------------Time to First Token----------------
Mean TTFT (ms):                          815107.71 
Median TTFT (ms):                        827756.68 
P99 TTFT (ms):                           1662652.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          153.73    
Median TPOT (ms):                        154.43    
P99 TPOT (ms):                           374.27    
---------------Inter-token Latency----------------
Mean ITL (ms):                           153.75    
Median ITL (ms):                         46.95     
P99 ITL (ms):                            62.72     
==================================================

Am i missing an important argument to include in vLLM?

Your current environment (if you think it is necessary)

Collecting environment information...
PyTorch version: 2.7.0+cu126
Is debug build: False
CUDA used to build PyTorch: 12.6
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2023.6.20250303 (x86_64)
GCC version: (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.34

Python version: 3.11.11 (main, Mar 17 2025, 21:02:09) [Clang 20.1.0 ] (64-bit runtime)
Python platform: Linux-6.1.129-138.220.amzn2023.x86_64-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A10G
GPU 1: NVIDIA A10G
GPU 2: NVIDIA A10G
GPU 3: NVIDIA A10G

Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R32
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             5600.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            768 KiB (24 instances)
L1i cache:                            768 KiB (24 instances)
L2 cache:                             12 MiB (24 instances)
L3 cache:                             96 MiB (6 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-47
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flake8==7.2.0
[pip3] numpy==2.2.5
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0
[pip3] torchao==0.10.0
[pip3] torchaudio==2.7.0
[pip3] torchvision==0.22.0
[pip3] transformers==4.51.1
[pip3] triton==3.3.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.5.dev659+g12e6c0b41 (git sha: 12e6c0b41)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     PHB     PHB     0-47    0               N/A
GPU1    PHB      X      PHB     PHB     0-47    0               N/A
GPU2    PHB     PHB      X      PHB     0-47    0               N/A
GPU3    PHB     PHB     PHB      X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

sglang version == Version: 0.4.6.post2

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xiakj]

mark

[thesillystudent]

sorry I did not understand?

[xiakj]

sorry I did not understand?

hh, I am curious about your question, so I want to keep looking this issue.

[Leon-Sander]

Try vllm v1 engine without FLASHINFER

[thesillystudent]

with flash attn vLLM

============ Serving Benchmark Result ============
Successful requests:                     50        
Benchmark duration (s):                  7893.84   
Total input tokens:                      1250000   
Total generated tokens:                  51200     
Request throughput (req/s):              0.01      
Output token throughput (tok/s):         6.49      
Total Token throughput (tok/s):          164.84    
---------------Time to First Token----------------
Mean TTFT (ms):                          3195140.80
Median TTFT (ms):                        3486283.59
P99 TTFT (ms):                           6978772.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1095.18   
Median TPOT (ms):                        1101.49   
P99 TPOT (ms):                           1329.01   
---------------Inter-token Latency----------------
Mean ITL (ms):                           1095.18   
Median ITL (ms):                         1000.87   
P99 ITL (ms):                            2425.86   
==================================================

[beaulian]

mark

[jinzhen-lin]

Could you try a old version of vllm (for example, v0.8.4).

I recently updated the marlin kernel, it had better performance in my test. But based on your testing, I suspect it may perform poorly in certain special situations.

[jinzhen-lin]

BTW, did you use the same python virtual env when you ran vllm and sglang ?

[thesillystudent]

@jinzhen-lin new env with 0.8.4

PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2023.6.20250303 (x86_64)
GCC version: (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.34

Python version: 3.11.11 (main, Mar 17 2025, 21:02:09) [Clang 20.1.0 ] (64-bit runtime)
Python platform: Linux-6.1.129-138.220.amzn2023.x86_64-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A10G
GPU 1: NVIDIA A10G
GPU 2: NVIDIA A10G
GPU 3: NVIDIA A10G

Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R32
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             5600.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            768 KiB (24 instances)
L1i cache:                            768 KiB (24 instances)
L2 cache:                             12 MiB (24 instances)
L3 cache:                             96 MiB (6 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-47
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flake8==7.2.0
[pip3] numpy==2.2.5
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchao==0.10.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.1
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.4
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     PHB     PHB     0-47    0               N/A
GPU1    PHB      X      PHB     PHB     0-47    0               N/A
GPU2    PHB     PHB      X      PHB     0-47    0               N/A
GPU3    PHB     PHB     PHB      X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.4/lib:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4:/usr/local/cuda-12.4/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

command

python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-32B-AWQ --port 8000 --gpu-memory-utilization 0.90 --tensor-parallel-size 4  --disable-log-requests --quantization awq_marlin --max-model-len 32768

getting OOM now

(VllmWorker rank=2 pid=1739536) INFO 05-15 17:46:31 [gpu_model_runner.py:1626] Graph capturing finished in 78 secs, took 2.28 GiB
(VllmWorker rank=3 pid=1739730) INFO 05-15 17:46:31 [gpu_model_runner.py:1626] Graph capturing finished in 78 secs, took 2.28 GiB
(VllmWorker rank=1 pid=1739270) INFO 05-15 17:46:31 [gpu_model_runner.py:1626] Graph capturing finished in 78 secs, took 2.28 GiB
(VllmWorker rank=0 pid=1739092) INFO 05-15 17:46:31 [gpu_model_runner.py:1626] Graph capturing finished in 78 secs, took 2.28 GiB
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380] Traceback (most recent call last):
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1484, in _dummy_sampler_run
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     sampler_output = self.model.sample(
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]                      ^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/model_executor/models/qwen3.py", line 319, in sample
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/v1/sample/sampler.py", line 49, in forward
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     sampled = self.sample(logits, sampling_metadata)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/v1/sample/sampler.py", line 115, in sample
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     random_sampled = self.topk_topp_sampler(
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]                      ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 98, in forward_native
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     logits = apply_top_k_top_p(logits, k, p)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]   File "/mnt/efs/env-i-034a02213795a8aa4/lib/python3.11/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 186, in apply_top_k_top_p
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]     logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1739270) ERROR 05-15 17:46:32 [multiproc_executor.py:380] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 446.00 MiB. GPU 1 has a total capacity of 22.09 GiB of which 307.44 MiB is free. Including non-PyTorch memory, this process has 21.78 GiB memory in use. Of the allocated memory 18.94 GiB is allocated by PyTorch, with 76.00 MiB allocated in private pools (e.g., CUDA Graphs), and 224.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[thesillystudent]

even fails for max-model-len of 100

python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-32B-AWQ --port 8000 --gpu-memory-utilization 0.90 --tensor-parallel-size 4  --disable-log-requests --quantization awq_marlin --max-model-len 100

[jinzhen-lin]

I'm not familiar with this OOM error, but if it's an issue with the Marlin kernel, it should affect other models as well. Could you try using a smaller model, such as Qwen/Qwen2.5-14B-Instruct-AWQ?

[beaulian]

Hi, @jinzhen-lin. I reproduct the same TTFT gap between vllm and sglang on 4xH20 with Qwen3-32B.

vllm 0.8.5.post1
sglang v0.4.6
cuda 12.4
flashinfer-python 0.2.5+cu124torch2.6
torch 2.6.0

vllm command

python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-32B --tensor-parallel-size 4  --disable-log-requests -O3

python benchmarks/benchmark_serving.py --backend vllm --model Qwen/Qwen3-32B --dataset-name random --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --random-input-len 25000 --random-output-len 1024 --num-prompts 100 --request-rate inf

result

============ Serving Benchmark Result ============
Successful requests: 100
Benchmark duration (s): 553.00
Total input tokens: 2500000
Total generated tokens: 71584
Request throughput (req/s): 0.18
Output token throughput (tok/s): 129.45
Total Token throughput (tok/s): 4650.27
---------------Time to First Token----------------
Mean TTFT (ms): 249534.77
Median TTFT (ms): 275350.85
P99 TTFT (ms): 494575.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 366.38
Median TPOT (ms): 256.60
P99 TPOT (ms): 1454.63
---------------Inter-token Latency----------------
Mean ITL (ms): 217.21
Median ITL (ms): 56.19
P99 ITL (ms): 1674.62
==================================================

sglang command

python -m sglang.launch_server --model-path Qwen/Qwen3-32B --tp-size 4 --attention-backend fa3 --enable-torch-compile

python -m sglang.bench_serving --backend sglang --model Qwen/Qwen3-32B --dataset-name random --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --random-input-len 25000 --random-output-len 1024 --num-prompts 100

result

============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max reqeuest concurrency: not set
Successful requests: 100
Benchmark duration (s): 255.88
Total input tokens: 1152120
Total generated tokens: 50346
Total generated tokens (retokenized): 50346
Request throughput (req/s): 0.39
Input token throughput (tok/s): 4502.63
Output token throughput (tok/s): 196.76
Total token throughput (tok/s): 4699.39
Concurrency: 89.29
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 228467.78
Median E2E Latency (ms): 234373.60
---------------Time to First Token----------------
Mean TTFT (ms): 70766.53
Median TTFT (ms): 56157.43
P99 TTFT (ms): 191493.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 650.61
Median TPOT (ms): 318.74
P99 TPOT (ms): 5999.80
---------------Inter-Token Latency----------------
Mean ITL (ms): 313.86
Median ITL (ms): 103.28
P95 ITL (ms): 121.32
P99 ITL (ms): 2587.42
Max ITL (ms): 159042.30
==================================================

[jinzhen-lin]

Then it’s probably not an issue related to the Marlin kernel. The bug might be coming from somewhere else. Unfortunately, I’m unable to offer any help on this.