InFlightBatching seems not working

[larme]

System Info

• CPU: amd64
• OS: Debian 12
• GPU: nvidia rtx4000 ada
• GPU driver: 535.161
• TensorRT-LLM version: 0.8
• tensorrtllm_backend version: 0.8

Who can help?

@kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Followed exactly steps of: https://developer.nvidia.com/blog/turbocharging-meta-llama-3-performance-with-nvidia-tensorrt-llm-and-nvidia-triton-inference-server/. The only change is setting kv_cache_free_gpu_mem_fraction=0.95.

Expected behavior

Then I run 2 copies of https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/inflight_batcher_llm/client/end_to_end_grpc_client.py in the same time. The first script will finish within 30 seconds. I expect the second one will finish around the same time (about 30 seconds)

actual behavior

The second will only finish after 60 seconds. Hence it seems like the batching is not working and every request will block other requests come after it.

additional notes

Very similar to #189, but the user reported that his issue was fixed after 0.6.1.

[byshiue]

I am not sure how do you run the end_to_end_grpc_client.py. Could you try

python3 scripts/launch_triton_server.py --world_size 1 --model_repo=llama_ifb/ --log

echo '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": "", "pad_id": 2, "end_id": 2}' > tmp.txt
printf '%s\n' {1..20} | xargs -I % -P 20 curl -X POST localhost:8000/v2/models/ensemble/generate -d @tmp.txt

and check the Active Request Count in the triton_log.txt?

Also, I see you are using the RTX 4000, which only has 8GB RAM. How do you follow the guid to run LLaMA 8B model?

[RunningLeon]

@byshiue hi, it seems that I met the same issue as in #453 . Could you take a look at the issue? THX.

[byshiue]

The issue is closed. You could also check the --max_num_tokens when build trt engine.