[New Model]: deepseek-vl2

[ZTurboX]

The model to consider.

support deepseek-vl2:https://huggingface.co/deepseek-ai/deepseek-vl2

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

It's already supported since #11578. Do you have issues using it?

[DarkLight1337]

cc @Isotr0py

[ZTurboX]

It's already supported since #11578. Do you have issues using it?

I use this command:
CUDA_VISIBLE_DEVICES=1,2 python -m vllm.entrypoints.openai.api_server --tensor-parallel-size 2 --model /data/models/deepseek-vl2 --served-model-name deepseek-vl2 --max-model-len 4096 --port 9030 --hf_overrides {"architectures": ["DeepseekVLV2ForCausalLM"]}

show an error: --hf_overrides: command not found

vllm version: 0.7.2

[DarkLight1337]

You need to escape the JSON string passed to --hf-overrides

[ZTurboX]

You need to escape the JSON string passed to --hf-overrides

has the same error:
CUDA_VISIBLE_DEVICES=1,2 python -m vllm.entrypoints.openai.api_server --tensor-parallel-size 2 --model /data/models/deepseek-vl2 --served-model-name deepseek-vl2 --max-model-len 4096 --port 9030 --hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'--chat-template /data/models/deepseek-vl2/template_deepseek_vl2.jinja

[DarkLight1337]

I think you accidentally added a space between - and m in python -m

[chenyzh28]

@DarkLight1337 Hi, I met some problem when I use vllm to run deepseek-vl2, 14421, the output does not look normal. I would be grateful if you could help me take a look.

[tunglinwood]

It's already supported since #11578. Do you have issues using it?
I use this command: CUDA_VISIBLE_DEVICES=1,2 python -m vllm.entrypoints.openai.api_server --tensor-parallel-size 2 --model /data/models/deepseek-vl2 --served-model-name deepseek-vl2 --max-model-len 4096 --port 9030 --hf_overrides {"architectures": ["DeepseekVLV2ForCausalLM"]}
show an error: --hf_overrides: command not found
vllm version: 0.7.2

@chenyzh28 @ZTurboX Please refer to #14105 where I have left my command line. And I can already tell that you should use "-" instead of "_" in --hf-overrides please kindly check again. Thank you.