[User] GGML_ASSERT failure for opencl

[rhogenson]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I am trying to use the WizardCoder Python 34B model with llama.cpp, using the opencl drivers. I am using a radeon 7900 XT.

Current Behavior

Llama.cpp crashes with the following output:

GGML_ASSERT: /build/rn9i4s34d56xdfq7njrkj416w9g08hh8-source/ggml.c:11236: ne02 == ne12
[alissa:29711] *** Process received signal ***
[alissa:29711] Signal: Aborted (6)
[alissa:29711] Signal code:  (-6)
[alissa:29711] [ 0] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x38d30)[0x7f7821372d30]
[alissa:29711] [ 1] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x87a8c)[0x7f78213c1a8c]
[alissa:29711] [ 2] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(gsignal+0x16)[0x7f7821372c86]
[alissa:29711] [ 3] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(abort+0xd7)[0x7f782135c8ba]
[alissa:29711] [ 4] /nix/store/5lcyfbikkjkgigv5kkvq77ssln40s90f-llama.cpp/lib/libllama.so(+0x3ccc7)[0x7f78218eccc7]
[alissa:29711] [ 5] /nix/store/5lcyfbikkjkgigv5kkvq77ssln40s90f-llama.cpp/lib/libllama.so(+0x4fab6)[0x7f78218ffab6]
[alissa:29711] [ 6] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x85dd4)[0x7f78213bfdd4]
[alissa:29711] [ 7] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x1079b0)[0x7f78214419b0]
[alissa:29711] *** End of error message ***
fish: Job 1, 'nix run github:ggerganov/llama.…' terminated by signal  (### Instruction:)
fish: Job hello, '' terminated by signal ### Response:' (SIGABRT)
fish: Job Abort, '' terminated by signal  ()

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 9 5900X 12-Core Processor
    CPU family:          25
    Model:               33
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           1
    Stepping:            2
    Frequency boost:     disabled
    CPU(s) scaling MHz:  71%
    CPU max MHz:         4950.1948
    CPU min MHz:         2200.0000
    BogoMIPS:            7386.25
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse
                         36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rd
                         tscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aper
                         fmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic
                          movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapi
                         c cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt 
                         tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_
                         l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 av
                         x2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb s
                         ha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_to
                         tal cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lb
                         rv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists 
                         pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pk
                         u ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   384 KiB (12 instances)
  L1i:                   384 KiB (12 instances)
  L2:                    6 MiB (12 instances)
  L3:                    64 MiB (2 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-23
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Mitigation; safe RET, no microcode
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, R
                         SB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

• Operating System, e.g. for Linux:

$ uname -a
Linux alissa 6.5.0 #1-NixOS SMP PREEMPT_DYNAMIC Sun Aug 27 21:49:51 UTC 2023 x86_64 GNU/Linux

• SDK version, e.g. for Linux:

I'm building from the latest flake.nix file.

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

This failure is reproducible with the WizardCoder Python 34B model, q4_K_M quantization downloaded from here: https://huggingface.co/TheBloke/WizardCoder-Python-34B-V1.0-GGUF

I used the following command to build and run:

nix run github:ggerganov/llama.cpp#opencl -- -m models/WizardCoder-Python-34B-V1.0/ggml-model-q4_K.gguf --temp 0 -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
hello

### Response:'

It should also work to compile normally with openCL support. I can see that the GGML_ASSERT failure is happening inside openCL-specific code, so it's important that this is the openCL version.

Failure Logs

$ nix run github:ggerganov/llama.cpp#opencl -- -m models/WizardCoder-Python-34B-V1.0/ggml-model-q4_K.gguf --temp 0 -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
hello

### Response:'
Log start
main: build = 0 (unknown)
main: seed  = 1693800992
ggml_opencl: selecting platform: 'AMD Accelerated Parallel Processing'
ggml_opencl: selecting device: 'gfx1100'
ggml_opencl: device FP16 support: true
llama_model_loader: loaded meta data with 20 key-value pairs and 435 tensors from /mnt/home/rose/llama.cpp/models/WizardCoder-Python-34B-V1.0/ggml-model-q4_K.gguf (version GGUF V2 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q4_K     [  8192, 32001,     1,     1 ]
llama_model_loader: - tensor    1:              blk.0.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor    2:              blk.0.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor    3:              blk.0.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor    4:         blk.0.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor    5:            blk.0.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor    6:              blk.0.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor    7:            blk.0.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor    8:           blk.0.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor    9:            blk.0.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   10:              blk.1.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   11:              blk.1.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   12:              blk.1.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   13:         blk.1.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   14:            blk.1.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   15:              blk.1.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   16:            blk.1.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   17:           blk.1.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   18:            blk.1.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   19:              blk.2.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   20:              blk.2.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   21:              blk.2.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   22:         blk.2.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   23:            blk.2.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   24:              blk.2.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   25:            blk.2.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   26:           blk.2.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   27:            blk.2.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   28:              blk.3.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   29:              blk.3.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   30:              blk.3.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   31:         blk.3.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   32:            blk.3.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   33:              blk.3.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   34:            blk.3.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   35:           blk.3.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   36:            blk.3.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   37:              blk.4.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   38:              blk.4.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   39:              blk.4.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   40:         blk.4.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   41:            blk.4.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   42:              blk.4.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   43:            blk.4.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   44:           blk.4.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   45:            blk.4.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   46:              blk.5.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   47:              blk.5.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   48:              blk.5.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   49:         blk.5.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   50:            blk.5.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   51:              blk.5.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   52:            blk.5.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   53:           blk.5.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   54:            blk.5.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   55:              blk.6.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   56:              blk.6.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   57:              blk.6.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   58:         blk.6.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   59:            blk.6.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   60:              blk.6.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   61:            blk.6.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   62:           blk.6.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   63:            blk.6.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   64:              blk.7.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   65:              blk.7.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   66:              blk.7.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   67:         blk.7.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   68:            blk.7.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   69:              blk.7.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   70:            blk.7.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   71:           blk.7.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   72:            blk.7.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   73:              blk.8.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   74:              blk.8.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   75:              blk.8.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   76:         blk.8.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   77:            blk.8.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   78:              blk.8.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   79:            blk.8.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   80:           blk.8.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   81:            blk.8.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   82:              blk.9.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   83:              blk.9.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   84:              blk.9.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   85:         blk.9.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   86:            blk.9.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   87:              blk.9.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   88:            blk.9.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   89:           blk.9.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   90:            blk.9.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   91:             blk.10.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   92:             blk.10.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   93:             blk.10.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor   94:        blk.10.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor   95:           blk.10.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   96:             blk.10.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor   97:           blk.10.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor   98:          blk.10.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor   99:           blk.10.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  100:             blk.11.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  101:             blk.11.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  102:             blk.11.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  103:        blk.11.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  104:           blk.11.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  105:             blk.11.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  106:           blk.11.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  107:          blk.11.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  108:           blk.11.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  109:             blk.12.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  110:             blk.12.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  111:             blk.12.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  112:        blk.12.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  113:           blk.12.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  114:             blk.12.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  115:           blk.12.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  116:          blk.12.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  117:           blk.12.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  118:             blk.13.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  119:             blk.13.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  120:             blk.13.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  121:        blk.13.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  122:           blk.13.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  123:             blk.13.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  124:           blk.13.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  125:          blk.13.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  126:           blk.13.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  127:             blk.14.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  128:             blk.14.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  129:             blk.14.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  130:        blk.14.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  131:           blk.14.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  132:             blk.14.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  133:           blk.14.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  134:          blk.14.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  135:           blk.14.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  136:             blk.15.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  137:             blk.15.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  138:             blk.15.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  139:        blk.15.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  140:           blk.15.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  141:             blk.15.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  142:           blk.15.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  143:          blk.15.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  144:           blk.15.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  145:             blk.16.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  146:             blk.16.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  147:             blk.16.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  148:        blk.16.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  149:           blk.16.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  150:             blk.16.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  151:           blk.16.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  152:          blk.16.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  153:           blk.16.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  154:             blk.17.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  155:             blk.17.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  156:             blk.17.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  157:        blk.17.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  158:           blk.17.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  159:             blk.17.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  160:           blk.17.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  161:          blk.17.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  162:           blk.17.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  163:             blk.18.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  164:             blk.18.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  165:             blk.18.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  166:        blk.18.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  167:           blk.18.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  168:             blk.18.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  169:           blk.18.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  170:          blk.18.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  171:           blk.18.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  172:             blk.19.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  173:             blk.19.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  174:             blk.19.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  175:        blk.19.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  176:           blk.19.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  177:             blk.19.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  178:           blk.19.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  179:          blk.19.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  180:           blk.19.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  181:             blk.20.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  182:             blk.20.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  183:             blk.20.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  184:        blk.20.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  185:           blk.20.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  186:             blk.20.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  187:           blk.20.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  188:          blk.20.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  189:           blk.20.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  190:             blk.21.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  191:             blk.21.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  192:             blk.21.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  193:        blk.21.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  194:           blk.21.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  195:             blk.21.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  196:           blk.21.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  197:          blk.21.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  198:           blk.21.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  199:             blk.22.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  200:             blk.22.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  201:             blk.22.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  202:        blk.22.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  203:           blk.22.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  204:             blk.22.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  205:           blk.22.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  206:          blk.22.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  207:           blk.22.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  208:             blk.23.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  209:             blk.23.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  210:             blk.23.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  211:        blk.23.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  212:           blk.23.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  213:             blk.23.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  214:           blk.23.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  215:          blk.23.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  216:           blk.23.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  217:             blk.24.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  218:             blk.24.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  219:             blk.24.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  220:        blk.24.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  221:           blk.24.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  222:             blk.24.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  223:           blk.24.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  224:          blk.24.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  225:           blk.24.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  226:             blk.25.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  227:             blk.25.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  228:             blk.25.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  229:        blk.25.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  230:           blk.25.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  231:             blk.25.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  232:           blk.25.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  233:          blk.25.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  234:           blk.25.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  235:             blk.26.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  236:             blk.26.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  237:             blk.26.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  238:        blk.26.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  239:           blk.26.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  240:             blk.26.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  241:           blk.26.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  242:          blk.26.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  243:           blk.26.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  244:             blk.27.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  245:             blk.27.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  246:             blk.27.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  247:        blk.27.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  248:           blk.27.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  249:             blk.27.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  250:           blk.27.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  251:          blk.27.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  252:           blk.27.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  253:             blk.28.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  254:             blk.28.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  255:             blk.28.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  256:        blk.28.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  257:           blk.28.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  258:             blk.28.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  259:           blk.28.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  260:          blk.28.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  261:           blk.28.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  262:             blk.29.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  263:             blk.29.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  264:             blk.29.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  265:        blk.29.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  266:           blk.29.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  267:             blk.29.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  268:           blk.29.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  269:          blk.29.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  270:           blk.29.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  271:             blk.30.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  272:             blk.30.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  273:             blk.30.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  274:        blk.30.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  275:           blk.30.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  276:             blk.30.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  277:           blk.30.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  278:          blk.30.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  279:           blk.30.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  280:             blk.31.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  281:             blk.31.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  282:             blk.31.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  283:        blk.31.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  284:           blk.31.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  285:             blk.31.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  286:           blk.31.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  287:          blk.31.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  288:           blk.31.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  289:             blk.32.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  290:             blk.32.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  291:             blk.32.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  292:        blk.32.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  293:           blk.32.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  294:             blk.32.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  295:           blk.32.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  296:          blk.32.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  297:           blk.32.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  298:             blk.33.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  299:             blk.33.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  300:             blk.33.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  301:        blk.33.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  302:           blk.33.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  303:             blk.33.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  304:           blk.33.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  305:          blk.33.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  306:           blk.33.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  307:             blk.34.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  308:             blk.34.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  309:             blk.34.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  310:        blk.34.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  311:           blk.34.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  312:             blk.34.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  313:           blk.34.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  314:          blk.34.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  315:           blk.34.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  316:             blk.35.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  317:             blk.35.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  318:             blk.35.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  319:        blk.35.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  320:           blk.35.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  321:             blk.35.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  322:           blk.35.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  323:          blk.35.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  324:           blk.35.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  325:             blk.36.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  326:             blk.36.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  327:             blk.36.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  328:        blk.36.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  329:           blk.36.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  330:             blk.36.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  331:           blk.36.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  332:          blk.36.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  333:           blk.36.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  334:             blk.37.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  335:             blk.37.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  336:             blk.37.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  337:        blk.37.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  338:           blk.37.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  339:             blk.37.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  340:           blk.37.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  341:          blk.37.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  342:           blk.37.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  343:             blk.38.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  344:             blk.38.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  345:             blk.38.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  346:        blk.38.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  347:           blk.38.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  348:             blk.38.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  349:           blk.38.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  350:          blk.38.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  351:           blk.38.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  352:             blk.39.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  353:             blk.39.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  354:             blk.39.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  355:        blk.39.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  356:           blk.39.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  357:             blk.39.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  358:           blk.39.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  359:          blk.39.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  360:           blk.39.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  361:             blk.40.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  362:             blk.40.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  363:             blk.40.attn_v.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  364:        blk.40.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  365:           blk.40.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  366:             blk.40.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  367:           blk.40.ffn_down.weight q4_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  368:          blk.40.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  369:           blk.40.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  370:             blk.41.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  371:             blk.41.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  372:             blk.41.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  373:        blk.41.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  374:           blk.41.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  375:             blk.41.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  376:           blk.41.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  377:          blk.41.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  378:           blk.41.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  379:             blk.42.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  380:             blk.42.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  381:             blk.42.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  382:        blk.42.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  383:           blk.42.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  384:             blk.42.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  385:           blk.42.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  386:          blk.42.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  387:           blk.42.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  388:             blk.43.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  389:             blk.43.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  390:             blk.43.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  391:        blk.43.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  392:           blk.43.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  393:             blk.43.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  394:           blk.43.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  395:          blk.43.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  396:           blk.43.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  397:             blk.44.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  398:             blk.44.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  399:             blk.44.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  400:        blk.44.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  401:           blk.44.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  402:             blk.44.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  403:           blk.44.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  404:          blk.44.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  405:           blk.44.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  406:             blk.45.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  407:             blk.45.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  408:             blk.45.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  409:        blk.45.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  410:           blk.45.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  411:             blk.45.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  412:           blk.45.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  413:          blk.45.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  414:           blk.45.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  415:             blk.46.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  416:             blk.46.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  417:             blk.46.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  418:        blk.46.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  419:           blk.46.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  420:             blk.46.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  421:           blk.46.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  422:          blk.46.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  423:           blk.46.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  424:             blk.47.attn_q.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  425:             blk.47.attn_k.weight q4_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  426:             blk.47.attn_v.weight q6_K     [  8192,  1024,     1,     1 ]
llama_model_loader: - tensor  427:        blk.47.attn_output.weight q4_K     [  8192,  8192,     1,     1 ]
llama_model_loader: - tensor  428:           blk.47.ffn_gate.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  429:             blk.47.ffn_up.weight q4_K     [  8192, 22016,     1,     1 ]
llama_model_loader: - tensor  430:           blk.47.ffn_down.weight q6_K     [ 22016,  8192,     1,     1 ]
llama_model_loader: - tensor  431:          blk.47.attn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  432:           blk.47.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  433:               output_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - tensor  434:                    output.weight q6_K     [  8192, 32001,     1,     1 ]
llama_model_loader: - kv   0:                       general.architecture str     
llama_model_loader: - kv   1:                               general.name str     
llama_model_loader: - kv   2:                       llama.context_length u32     
llama_model_loader: - kv   3:                     llama.embedding_length u32     
llama_model_loader: - kv   4:                          llama.block_count u32     
llama_model_loader: - kv   5:                  llama.feed_forward_length u32     
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32     
llama_model_loader: - kv   7:                 llama.attention.head_count u32     
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32     
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32     
llama_model_loader: - kv  10:                       llama.rope.freq_base f32     
llama_model_loader: - kv  11:                          general.file_type u32     
llama_model_loader: - kv  12:                       tokenizer.ggml.model str     
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr     
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr     
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr     
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32     
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32     
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32     
llama_model_loader: - kv  19:               general.quantization_version u32     
llama_model_loader: - type  f32:   97 tensors
llama_model_loader: - type q4_K:  289 tensors
llama_model_loader: - type q6_K:   49 tensors
llm_load_print_meta: format         = GGUF V2 (latest)
llm_load_print_meta: arch           = llama
llm_load_print_meta: vocab type     = SPM
llm_load_print_meta: n_vocab        = 32001
llm_load_print_meta: n_merges       = 0
llm_load_print_meta: n_ctx_train    = 16384
llm_load_print_meta: n_ctx          = 512
llm_load_print_meta: n_embd         = 8192
llm_load_print_meta: n_head         = 64
llm_load_print_meta: n_head_kv      = 8
llm_load_print_meta: n_layer        = 48
llm_load_print_meta: n_rot          = 128
llm_load_print_meta: n_gqa          = 8
llm_load_print_meta: f_norm_eps     = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: n_ff           = 22016
llm_load_print_meta: freq_base      = 1000000.0
llm_load_print_meta: freq_scale     = 1
llm_load_print_meta: model type     = 34B
llm_load_print_meta: model ftype    = mostly Q4_K - Medium
llm_load_print_meta: model size     = 33.74 B
llm_load_print_meta: general.name   = models
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.14 MB
llm_load_tensors: using OpenCL for GPU acceleration
llm_load_tensors: mem required  = 19282.63 MB (+   96.00 MB per state)
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/49 layers to GPU
llm_load_tensors: VRAM used: 0 MB
...................................................................................................
llama_new_context_with_model: kv self size  =   96.00 MB
llama_new_context_with_model: compute buffer total size =  119.47 MB

system_info: n_threads = 12 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | 
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.000000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0


 Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
hello

### Response:GGML_ASSERT: /build/rn9i4s34d56xdfq7njrkj416w9g08hh8-source/ggml.c:11236: ne02 == ne12
[alissa:29711] *** Process received signal ***
[alissa:29711] Signal: Aborted (6)
[alissa:29711] Signal code:  (-6)
[alissa:29711] [ 0] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x38d30)[0x7f7821372d30]
[alissa:29711] [ 1] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x87a8c)[0x7f78213c1a8c]
[alissa:29711] [ 2] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(gsignal+0x16)[0x7f7821372c86]
[alissa:29711] [ 3] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(abort+0xd7)[0x7f782135c8ba]
[alissa:29711] [ 4] /nix/store/5lcyfbikkjkgigv5kkvq77ssln40s90f-llama.cpp/lib/libllama.so(+0x3ccc7)[0x7f78218eccc7]
[alissa:29711] [ 5] /nix/store/5lcyfbikkjkgigv5kkvq77ssln40s90f-llama.cpp/lib/libllama.so(+0x4fab6)[0x7f78218ffab6]
[alissa:29711] [ 6] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x85dd4)[0x7f78213bfdd4]
[alissa:29711] [ 7] /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6(+0x1079b0)[0x7f78214419b0]
[alissa:29711] *** End of error message ***
fish: Job 1, 'nix run github:ggerganov/llama.…' terminated by signal  (### Instruction:)
fish: Job hello, '' terminated by signal ### Response:' (SIGABRT)
fish: Job Abort, '' terminated by signal  ()

Environment info:

$ git log | head -1
commit cf9b08485c4c2d4d945c6e74fe20f273a38b6104

$ lscpu | egrep "AMD|Flags"
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen 9 5900X 12-Core Processor
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:                     AMD-V

$ python3 --version
Python 3.10.12

$ pip list | egrep "torch|numpy|sentencepiece"
numpy         1.24.0
sentencepiece 0.1.98

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip

$ make --version | head -1
GNU Make 4.4.1

$ md5sum ./models/WizardCoder-Python-34B-V1.0/ggml-model-q4_K.gguf
509293692a563a4eb6ee96230111e608  ./models/WizardCoder-Python-34B-V1.0/ggml-model-q4_K.gguf

[Happenedtostumblein]

Seeing issue as well. Only affects l2.

@rhogenson Are you getting this error with Llama 1 models too?

[KerfuffleV2]

Maybe related to #2736?

This isn't really a solution, but you could try compiling with ROCM instead of using OpenCL. It should also be considerably faster. I never really had much luck with OpenCL, but now that the ROCM support is merged it's both more reliable and much faster. edit: Yikes, ignore this.

[Happenedtostumblein]

Thanks @KerfuffleV2 FWIW I’m on Nvidia so not sure if ROCM is an option or if this could be related to 2736?

[KerfuffleV2]

Oh, apologies. I'm actually a moron, but the answer is still almost the same except now I'd say you should use CUDA! For all the same reasons.

[Happenedtostumblein]

No, you do not know moron. I have dreams about ggml being in python so I can understand it…

I may resort to CUDA, but I like that OpenCL is open source. What do you think the chances are of us figuring out the root of the issue?

Also, kind of off-topic but do you happen to have anything to say about this issue (leejet/stable-diffusion.cpp#48), also about OpenCL?

[KerfuffleV2]

What do you think the chances are of us figuring out the root of the issue?

I'm afraid the chance of me figuring it out are pretty low. As soon as the ROCM stuff got merged, I switched to that. I can't speak for anyone else, but it seems like the use case for it is pretty limited right now: People on AMD hardware will use ROCM, people on Nvidia will mostly use CUDA, people on Mac will use Metal. The only users that would still use OpenCL now are... people with Intel's GPUs? (And you, of course. :)

Also it seems like even when stuff is working the performance is considerably lower than options like CUDA or ROCM.

do you happen to have anything to say about this issue

Sorry, I don't really know anything about that.

[Happenedtostumblein]

Thank you for taking time out of your day to talk 🥇

[rhogenson]

I haven't tested this with a llama 1 model, so I can't say whether this issue affects llama 1 as well. However I did some limited testing with other quantization levels, and it seems like it only breaks with the new k-quant quantizations. I didn't do much testing though, so that might not be true.

Unfortunately NixOS doesn't support ROCM version 5.6 yet. We're still on 5.4. It should work on ROCM, as this assertion failure happens inside openCL-only code.

Digging into the code, it looks like this is the result of a TODO that was never completed. The GGML openCL driver doesn't support matrix multiplication when the 2nd or 3rd dimensions are not the same. I was able to work around this issue by skipping the openCL matrix multiplication in this case, and falling back to the slow CPU matrix multiplication. It's pretty slow though, so a real fix would need to implement this missing feature for the openCL version.

[Happenedtostumblein]

Can you point to a line number where it’s “doing this, but it should be doing that”…or is it not that straightforward?

[rhogenson]

It's simple-ish, but it's been a while since I took linear algebra 😅

Take a look at line 11234 of ggml.c, it has this TODO:

// TODO: handle case when src0 is broadcast-able into src1 across 2nd,3rd dimension
//       ref: https://github.com/ggerganov/ggml/pull/224

And also look at the code for OpenBLAS just below. It had a similar assert removed in commit ef3f333, which added this broadcasting for the openblas implementation.

[shibe2]

I implemented what's needed for Llama 2 inference on CLBlast. However, I'm having issues with current llama.cpp code, and I cannot test it. So I ported my changes to an earlier version of the codebase that uses GGJT format.

Diff here. Note that it is on top of other changes that need review on their own. Full set of changes here.

I will port it back to current when the problems are resolved.

Update: My issues with inference may be related to RoPE scaling. I'm now downloading a model with normal 4096 context size.

[shibe2]

I got Llama 2 70B working and tested my implementation. Output at 0 temperature is slightly different between CPU and CLBlast builds, but both are okay. Prompt processing is significantly faster with CLBlast, even without ngl. With my relatively small VRAM, I get only marginal performance increase from ngl.

I'm still having issues with Code Llama.

My implementation is here: https://github.com/shibe2/llama.cpp/tree/cl-gqa
It is not yet ready for merging. I intend to keep that branch updated until final implementation is merged. If you want to try/use it, options are:

• build from my branch;
• apply changes on top of current master;
• replace ggml-opencl.cpp with one from my tree and remove relevant assertions from ggml.c.

[KerfuffleV2]

You might want to make a draft pull request rather than just commenting here. There's a better chance of people seeing it and giving feedback, it's kind of hidden away in the issue here currently.