How to use speculative decoding?

[irthomasthomas]

Hello,

I've read the docs and tried a few different ways to start speculative decoding, but they all fail.
E.g.
error: unrecognized arguments: --draft_model=prompt-lookup-decoding --draft_model_num_pred_tokens=2

or
Extra inputs are not permitted [type=extra_forbidden, input_value='prompt-lookup-decoding', input_type=str]

So what is the correct way to start an openai server with speculative decoding?

Cheers.

[abetlen]

Hey @irthomasthomas that should work, are you sure you have the latest version installed?

python3 -c "import llama_cpp; print(llama_cpp.__version__)"

If you run that the version should be at least 0.2.38 for speculative decoding support.

[irthomasthomas]

Thanks @abetlen, I forgot I was running multiple python environments... Anyway, after updating, I'm now getting a new problem: ValueError: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

Is that related to this? ggml-org/llama.cpp#5240

Here is the command I ran:
python3 -m llama_cpp.server --model /run/media/thomas/828ca7e5-381d-4149-8fc5-6d6aa26b90f2/Models/deepseek/deepseek-coder-33B-instruct-GGUF/deepseek-coder-33b-instruct.Q4_K_M.gguf --n_gpu_layers 56 --tensor_split 64 36 --offload_kqv false --n_ctx 8000 --n_batch 56 --chat_format chatml

[irthomasthomas]

I opened a new issue as its unrelated to this one. #1166