Misc. bug: Eval bug: Repetitive Output After Certain Token Count When Using -np > 1 in llama.cpp (Ver. b5468)

[thomasbergersen]

Name and Version

PS D:\llama.cpp\release\llama-b5468-bin-win-cuda-12.4-x64> ./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from D:\llama.cpp\release\llama-b5468-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\llama.cpp\release\llama-b5468-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\llama.cpp\release\llama-b5468-bin-win-cuda-12.4-x64\ggml-cpu-alderlake.dll
version: 5468 (d13d0f6)
built with clang version 18.1.8 for x86_64-pc-windows-msvc

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Problem description & steps to reproduce

Starting from commit b5434 and onward, setting the -np (or --n-parallel) parameter greater than 1 in llama.cpp causes the model to generate repetitive outputs — such as endlessly repeating characters like '=' or '3' — after a certain number of tokens have been decoded.

First Bad Commit

No response

Relevant log output

[ggerganov]

Could you give the command and model that you use and repro steps?

[ggerganov]

This seems to be the same problem as #13716. It's likely a regression from the SWA change in #13194.

So far I cannot reproduce on Mac, so I'm not sure what could be the root cause.

Could you run the following command and see if there are any repetitions generated:

llama-parallel -hf ggml-org/gemma-3-4b-it-GGUF -np 2 -ns 32 --top-k 1 --junk 132 -c 16384

If everything is good, the output should have random questions and answers like this:

[thomasbergersen]

Hi @ggerganov , Here are the commands I used for testing:

.\llama-server -m d:\models\Qwen2.5-Coder-14B-Instruct_IQ3_XS.gguf -fa -ngl 60 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 8192 --repeat-penalty 1.5 --jinja --host 0.0.0.0 --port 1130 --mlock --no-perf -np 5

When switching to other models (e.g. Qwen2.5-Coder-0.5B-Instruct) and also setting -np=2, I observed the same behavior after multiple rounds of large input — the model eventually starts producing repetitive outputs.

.\llama-server -m d:\models\Qwen2.5-Coder-0.5B-Instruct.gguf -fa -ngl 60 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 8192 --repeat-penalty 1.5 --jinja --host 0.0.0.0 --port 1130 --mlock --no-perf -np 5

Also, the llama-parallel binary was not included in the official release build. When I tried the following command:

llama-parallel -hf ggml-org/gemma-3-4b-it-GGUF -np 2 -ns 32 --top-k 1 --junk 132 -c 16384

It failed since --junk is not a valid parameter. I rebuilt the binary myself to continue testing.

./llama-parallel -m d:\models\Qwen2.5-Coder-14B-Instruct_IQ3_XS.gguf -np 2 -ns 32 --top-k 1 --junk 132 -c 16384

error: invalid argument: --junk

[ggerganov]

Hm, --junk has been added since b5414 (518329b), how come it is missing?

Do you observe the problem if you remove -fa?

Are you confident that b5429 specifically introduced the regression?

[thomasbergersen]

I’ve downloaded and tested the recent official release builds again. After comparison, I can confirm that the issue begins with commit b5434. The problem does not occur in b5432, which works as expected.

./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from D:\llama.cpp\release\llama-b5434-bin-win-cuda12.4-x64\ggml-cuda.dll  
load_backend: loaded RPC backend from D:\llama.cpp\release\llama-b5434-bin-win-cuda12.4-x64\ggml-rpc.dll    
load_backend: loaded CPU backend from D:\llama.cpp\release\llama-b5434-bin-win-cuda12.4-x64\ggml-cpu-alderlake.dll
version: 5434 (b69f1647)
built with MSVC 19.29.30159.0 for Windows AMD64

[JohannesGaessler]

Can you check whether #13742 fixes the issue?

[thomasbergersen]

Hi @ggerganov @JohannesGaessler,
I’ve confirmed that the repetitive output issue is fixed in commit b5471 and later.
Thank you for the great work!