StableLM inference triggers `GGML_ASSERT(n_embd_head == hparams.n_rot);`

[brittlewis12]

Thanks very much for your fantastic work with this library @ggerganov!

Updated to latest llama.cpp (revision: e790eef) this morning.

StableLM-Zephyr-3b quants triggering GGML_ASSERT(n_embd_head == hparams.n_rot); introduced in #4889 (f445c0e#diff-150dc86746a90bad4fc2c3334aeb9b5887b3adad3cc1459446717638605348efR5533).

It seems the values I see printed for the quants I have (Q4_K_M) are as follows:

llm_load_print_meta: n_rot            = 20
llm_load_print_meta: n_embd_head_k    = 80
llm_load_print_meta: n_embd_head_v    = 80

You mentioned chance of breaking Persimmon:

All models now will use the hparams.n_rot value instead of relying on a custom parameter (like n_embd_head). Both for ggml_rope_custom and llm_build_k_shift. I suspect this might break Persimmon inference, because I'm not sure if hparams.n_rot is correctly populated in the meta data of the model, but if that is the case, then it should be fixed.

Is this a similar issue to Persimmon where the model metadata is incorrect, or is this a distinct problem?

[ggerganov]

Can you confirm that it works if you comment the GGML_ASSERT on line 5533?

[brittlewis12]

Yes, it still appears to work great with just that change!

diff --git a/llama.cpp b/llama.cpp
index 29f8873f..98c0b783 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -5530,7 +5530,7 @@ struct llm_build_context {

         const int64_t n_embd_head = hparams.n_embd_head_v;
         GGML_ASSERT(n_embd_head == hparams.n_embd_head_k);
-        GGML_ASSERT(n_embd_head == hparams.n_rot);
+        // GGML_ASSERT(n_embd_head == hparams.n_rot);

         struct ggml_tensor * cur;
         struct ggml_tensor * inpL;

❯ ./stablezephyr.sh
Log start
main: build = 1841 (930f907d)
main: built with Apple clang version 15.0.0 (clang-1500.1.0.2.5) for arm64-apple-darwin22.6.0
main: seed  = 1705085316
llama_model_loader: loaded meta data with 21 key-value pairs and 356 tensors from /Users/tito/code/hugtug/stablelm-zephyr-3b.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = stablelm
llama_model_loader: - kv   1:                               general.name str              = source
llama_model_loader: - kv   2:                    stablelm.context_length u32              = 4096
llama_model_loader: - kv   3:                  stablelm.embedding_length u32              = 2560
llama_model_loader: - kv   4:                       stablelm.block_count u32              = 32
llama_model_loader: - kv   5:               stablelm.feed_forward_length u32              = 6912
llama_model_loader: - kv   6:              stablelm.rope.dimension_count u32              = 20
llama_model_loader: - kv   7:              stablelm.attention.head_count u32              = 32
llama_model_loader: - kv   8:             stablelm.use_parallel_residual bool             = true
llama_model_loader: - kv   9:      stablelm.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  11:                      tokenizer.ggml.tokens arr[str,50304]   = ["<|endoftext|>", "<|padding|>", "!",...
llama_model_loader: - kv  12:                  tokenizer.ggml.token_type arr[i32,50304]   = [3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  13:                      tokenizer.ggml.merges arr[str,50009]   = ["Ġ Ġ", "Ġ t", "Ġ a", "h e", "i n...
llama_model_loader: - kv  14:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 0
llama_model_loader: - kv  16:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  17:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}\n{% if m...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  130 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: mismatch in special tokens definition ( 31/50304 vs 52/50304 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = stablelm
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 50304
llm_load_print_meta: n_merges         = 50009
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 2560
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 20
llm_load_print_meta: n_embd_head_k    = 80
llm_load_print_meta: n_embd_head_v    = 80
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 2560
llm_load_print_meta: n_embd_v_gqa     = 2560
llm_load_print_meta: f_norm_eps       = 1.0e-05
llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 6912
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 3B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 2.80 B
llm_load_print_meta: model size       = 1.59 GiB (4.88 BPW)
llm_load_print_meta: general.name     = source
llm_load_print_meta: BOS token        = 0 '<|endoftext|>'
llm_load_print_meta: EOS token        = 0 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<|endoftext|>'
llm_load_print_meta: PAD token        = 0 '<|endoftext|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_tensors: ggml ctx size       =    0.14 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  1629.45 MiB, ( 1629.52 / 21845.34)
llm_load_tensors: system memory used  = 1627.87 MiB
............................................................................................
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/tito/code/llama.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 22906.50 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =  1280.00 MiB, ( 2910.20 / 21845.34)
llama_new_context_with_model: KV self size  = 1280.00 MiB, K (f16):  640.00 MiB, V (f16):  640.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, ( 2910.22 / 21845.34)
llama_build_graph: non-view tensors processed: 741/741
llama_new_context_with_model: compute buffer total size = 282.19 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   279.02 MiB, ( 3189.22 / 21845.34)

system_info: n_threads = 7 / 10 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
main: interactive mode on.
Reverse prompt: '<|user|>'
Reverse prompt: '<|assistant|>'
Reverse prompt: '<|endoftext|>'
Reverse prompt: '### Instruction:

'
Input prefix: '<|user|>\n'
Input suffix: '<|endoftext|>\n<|assistant|>\n'
sampling:
	repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
	top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.700
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp
generate: n_ctx = 4096, n_batch = 512, n_predict = -1, n_keep = 1


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.


<|user|>
> <|user|>\ntell me a joke
<|endoftext|>\n<|assistant|>\n
Sure, here's a joke for you:

Why don't scientists trust atoms?

Because they make up everything!

> <|user|>\n

[jquesnelle]

I think we also need to remove https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp#L5532 which is triggering for me after this fix. Can confirm it still works when that line is removed 🙂

[ggerganov]

Hm, which model is that - can you paste the llm_load_print_meta logs?

[jquesnelle]

Wait -- my build system didn't detect the first change and rebuild. Latest works great, false alarm, sorry 😂 !