Description
System Info
[libprotobuf ERROR /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/third_party/protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format inference.ModelConfig: 29:17: Expected integer, got: $
E0617 11:16:30.424716 31125 model_repository_manager.cc:1335] Poll failed for model directory 'tensorrt_llm': failed to read text proto from tensorrtllm_backend/triton_model_repo/tensorrt_llm/config.pbtxt
Who can help?
No response
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
https://github.com/triton-inference-server/tensorrtllm_backend/
Followed the official guide with llama3-8b chat model
Expected behavior
Loads all the model
Serves LLM with gRPC / https at 8000, 8001 ports
actual behavior
I0617 11:16:30.692886 31127 server.cc:307] Waiting for in-flight requests to complete.
I0617 11:16:30.692893 31127 server.cc:323] Timeout 30: Found 0 model versions that have in-flight inferences
I0617 11:16:30.692914 31127 server.cc:338] All models are stopped, unloading models
I0617 11:16:30.692919 31127 server.cc:347] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
error: creating server: Internal - failed to load all models
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[3135,1],1]
Exit code: 1
additional notes
W0617 11:16:30.423990 31125 server.cc:251] failed to enable peer access for some device pairs
[libprotobuf ERROR /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/third_party/protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format inference.ModelConfig: 29:17: Expected integer, got: $
E0617 11:16:30.424716 31125 model_repository_manager.cc:1335] Poll failed for model directory 'tensorrt_llm': failed to read text proto from tensorrtllm_backend/triton_model_repo/tensorrt_llm/config.pbtxt
I0617 11:16:30.424744 31125 server.cc:607]
Above in bolds seems to be the issue.
I don't know how to resolve above.
Please help.