Feature Request: Interleaved sliding window attention support for gemma 2 and 3

[ymcki]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

While trying to estimate how much KV cache uses at 128k context for the major models using formula and running llama.cpp empirically, I noticed that gemma 3 has an extremely high KV cache usage:

https://www.reddit.com/r/LocalLLaMA/comments/1jl33br/qwq32b_has_the_highest_kv_cachemodel_size_ratio/

When I read the gemma 3 technical report, its Figure 6 says it has a 5:1 interleaved sliding window attention that can reduce the KV cache usage to one sixth.

https://arxiv.org/html/2503.19786v1

I checked the llama.cpp code, it seems like while there is kq_mask for sliding window attention to speed up inference, there seems to be no code to reduce KV cache usage.

Motivation

I think it will be great if llama.cpp can support this feature such that the gemma models can be applicable to the long context situation.

Possible Implementation

No response

[h9j6k]

second this, this issues #12352 seems to be related and answered at the bottom

[Dampfinchen]

Yes, currently the Gemma models are really heavy to run due to the huge KV Cache. If this gets implemented, it would provide a major speedup.

[Vorlent]

Gemma 2 9B and Gemma 3 12B have a crazy wide head length of 256. This means that each attention head in Gemma 2 and 3 is twice as heavy in terms of memory per token than most 100B+ parameter models, assuming the same head_count_kv, which is 8 for LLama 4 Maverick, LLama 3.1 405B and the above Gemma models.

To make matters worse, there are no flash attention kernels in llama.cpp that support a head length of 256 and quantization at the same time so everyone is taking the hit of storing the full fp16 KV cache. Paradoxically it's better to use Gemma 3 27B instead, because that one only has a head size of 128 and should support KV quantization.

[ymcki]

Gemma 2 9B and Gemma 3 14B have a crazy wide head length of 256. This means that each attention head in Gemma 2 and 3 is twice as heavy in terms of memory per token than most 100B+ parameter models, assuming the same head_count_kv, which is 8 for LLama 4 Maverick, LLama 3.1 405B and the above Gemma models.

To make matters worse, there are no flash attention kernels in llama.cpp that support a head length of 256 and quantization at the same time so everyone is taking the hit of storing the full fp16 KV cache. Paradoxically it's better to use Gemma 3 27B instead, because that one only has a head size of 128 and should support KV quantization.

Thanks for telling us that a head_dim of 256 has no flash attention support.

The issue here is that since iSWA KV cache allocation is not supported, the fp16 KV cache of the 27B model will be 62GB at 128k context instead of 10.4GB. Since iSWA support is implemented in HF transformers and ollama, it would also be great if llama.cpp also supports it to unlock the potential of gemma 3 at long context.

[Vorlent]

Thanks for telling us that a head_dim of 256 has no flash attention support.

I mean there is, but only for fp16 (= heavy KV cache): https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/fattn-vec-f32.cuh#L428

[ymcki]

Thanks for telling us that a head_dim of 256 has no flash attention support.

I mean there is, but only for fp16 (= heavy KV cache): https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/fattn-vec-f32.cuh#L428

Thanks for your clarification. I also found that gemma 3 12b cannot run when I try to quantize the V cache because flash attention is required to quantize the V cache. But I can run by only quantizing the K cache.

[Xonar92]

I would also like some form of support for this. The reduction on VRAM requirements could potentially be huge!

[ymcki]

I would also like some form of support for this. The reduction on VRAM requirements could potentially be huge!

Yeah. For b27, it can reduce fp16 kv cache at 128k context from 62GB to 10.4GB. Making it viable to run on a 24GB card.

[Dampfinchen]

Gemma 2 9B and Gemma 3 14B have a crazy wide head length of 256. This means that each attention head in Gemma 2 and 3 is twice as heavy in terms of memory per token than most 100B+ parameter models, assuming the same head_count_kv, which is 8 for LLama 4 Maverick, LLama 3.1 405B and the above Gemma models.
To make matters worse, there are no flash attention kernels in llama.cpp that support a head length of 256 and quantization at the same time so everyone is taking the hit of storing the full fp16 KV cache. Paradoxically it's better to use Gemma 3 27B instead, because that one only has a head size of 128 and should support KV quantization.

Thanks for telling us that a head_dim of 256 has no flash attention support.

The issue here is that since iSWA KV cache allocation is not supported, the fp16 KV cache of the 27B model will be 62GB at 128k context instead of 10.4GB. Since iSWA support is implemented in HF transformers and ollama, it would also be great if llama.cpp also supports it to unlock the potential of gemma 3 at long context.

Wait, Ollama supports iSWA?

Ollama is like a parasite, only take from llama.cpp but never give back.

[ymcki]

Gemma 2 9B and Gemma 3 14B have a crazy wide head length of 256. This means that each attention head in Gemma 2 and 3 is twice as heavy in terms of memory per token than most 100B+ parameter models, assuming the same head_count_kv, which is 8 for LLama 4 Maverick, LLama 3.1 405B and the above Gemma models.
To make matters worse, there are no flash attention kernels in llama.cpp that support a head length of 256 and quantization at the same time so everyone is taking the hit of storing the full fp16 KV cache. Paradoxically it's better to use Gemma 3 27B instead, because that one only has a head size of 128 and should support KV quantization.

Thanks for telling us that a head_dim of 256 has no flash attention support.
The issue here is that since iSWA KV cache allocation is not supported, the fp16 KV cache of the 27B model will be 62GB at 128k context instead of 10.4GB. Since iSWA support is implemented in HF transformers and ollama, it would also be great if llama.cpp also supports it to unlock the potential of gemma 3 at long context.

Wait, Ollama supports iSWA?

Ollama is like a parasite, only take from llama.cpp but never give back.

I can run 27b can run 128k on a single 3090 with ollama. I also saw some code that seems related to iswa.

[dinerburger]

Their implementation does indeed appear to support iSWA.

[Dampfinchen]

Their implementation does indeed appear to support iSWA.

isnt it possible to Port it over?

[ymcki]

Of course u can be ported. It was also implemented in hf transformers way b4 ollama. But someone still need to do it.

It requires implememtation of rolling kv cache which is more complicated to do in c/c++ tham python and golang