Quantization of Llama3.2-1b/3b with QNN backend

[moxprox]

Hello executorch team

I've been exploring my options to quantize and export either Llama3.2-1B or 3B using the Executorch + QNN workflow. I have tried different approaches but none of them has worked out. My question is, given the current state of Executorch + QNN backend, is it possible to quantize a Llama3.2 1B/3B model that responds with a correct answer? For my application I'm interested in running the model on NPU so running on CPU is not an option. Could you also take a look at what I've tried and let me know if you have any suggestion I can follow.

Thanks in advance.

This is what I have tried:

1. Quantize and export the model with Executorch + QNN (examples/qualcomm/oss_scripts/llama/llama.py --ptq 16a4w) produces a model that returns invalid responses (symbols or non-words) during inferences. Seems like similar issues have been already raised.

2. Export a pre-quantized model such as Llama-3.2-1B-Instruct-SpinQuant_INT4_EO. For this I get an error "Only Tensors of floating point and complex dtype can require gradients". Can you confirm that this feature is not supported as suggested by this ticket?

python -m examples.models.llama.export_llama -t /workplace/models/Llama3.1-8B-Instruct/tokenizer.model -p /workplace/models/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8/params.json -c /workplace/models/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8/consolidated.00.pth --use_kv_cache --qnn --soc_model SM8550

3. I noticed there are no SpinQuant matrices for smaller models like 1B/3B however I tried generating one following this guide. I was able to generate a matrix file (R.bin) however when running the model on the target device I get the following failures I'm still investigating:

python -m examples.models.llama.export_llama -t /workplace/models/Llama-3.2-1B/original/tokenizer.model -p /workplace/models/Llama-3.2-1B/original/params.json -c /workplace/models/Llama-3.2-1B/original/consolidated.00.pth --use_kv_cache --qnn --pt2e_quantize qnn_16a4w --disable_dynamic_shape --num_sharding 8 --calibration_tasks wikitext --calibration_limit 1 --calibration_seq_length 128 --soc_model SM8550 --optimized_rotation_path /workplace/models/matrix_llama3.2-1b_w16-a4/R.bin --calibration_data "<|start_header_id|>system<|end_header_id|>\n\nYou are a funny chatbot.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nCould you tell me about Facebook?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Failed to create transport for device, error: 4000
[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Failed to load skel, error: 4000
[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Transport layer setup failed: 14001
[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Failed to parse default platform info: 14001
[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Failed to load default platform info: 14001
[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Failed to parse platform config: 14001
[ERROR] [Qnn ExecuTorch]: Failed to create device_handle for Backend ID 6, error=14001
E 00:00:01.580015 executorch:QnnManager.cpp:305] Fail to configure Qnn device
E 00:00:01.580019 executorch:QnnExecuTorchBackend.cpp:95] Fail to initialize Qnn Manager
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Backend 1 free cleanup called during process exit

cc @cccclai @cbilgin

[SS-JIA]

Looping in @cccclai who is most familiar with the QNN backend. Assigning to her for now.

[cccclai]

Are you able to run a simple non llama model (like https://github.com/pytorch/executorch/blob/main/examples/qualcomm/scripts/export_example.py) and run it on device?

[moxprox]

@cccclai thanks for looking into the issue. I have tried with add.pte and also deeplab. For both I'm able to see the following message:

I 00:00:00.157722 executorch:qnn_executor_runner.cpp:506] Model executed successfully.

This is the full output from executing deeplab model with no inputs:

sh-5.1# ./qnn_executor_runner --model_path dl3_qnn_q8.pte
I 00:00:00.000317 executorch:qnn_executor_runner.cpp:171] Model file dl3_qnn_q8.pte is loaded.
I 00:00:00.000359 executorch:qnn_executor_runner.cpp:181] Using method forward
I 00:00:00.000368 executorch:qnn_executor_runner.cpp:228] Setting up planned buffer 0, size 9031680.
[INFO] [Qnn ExecuTorch]: Deserializing processed data using QnnContextCustomProtocol
[INFO] [Qnn ExecuTorch]: QnnDlcManager Get Qnn Context blob bytes 61731624
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[WARNING] [Qnn ExecuTorch]:  <W> Initializing HtpProvider
[WARNING] [Qnn ExecuTorch]:  <W> Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[INFO] [Qnn ExecuTorch]: QnnContextCustomProtocol expected magic number: 0x5678abcd but get: 0x2000000
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/2.31.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.4.0/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[WARNING] [Qnn ExecuTorch]:  <W> Function not called, PrepareLib isn't loaded!
[WARNING] [Qnn ExecuTorch]:  <W> Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=1 optimization.
I 00:00:00.145747 executorch:qnn_executor_runner.cpp:252] Method loaded.
E 00:00:00.147047 executorch:method.cpp:1202] Output 0 is memory planned, or is a constant. Cannot override the existing data pointer.
I 00:00:00.147073 executorch:qnn_executor_runner.cpp:309] ignoring error from set_output_data_ptr(): 0x2
E 00:00:00.148151 executorch:method.cpp:1202] Output 1 is memory planned, or is a constant. Cannot override the existing data pointer.
I 00:00:00.148173 executorch:qnn_executor_runner.cpp:309] ignoring error from set_output_data_ptr(): 0x2
I 00:00:00.148179 executorch:qnn_executor_runner.cpp:315] Inputs prepared.
I 00:00:00.148582 executorch:qnn_executor_runner.cpp:497] Input list not provided. Inputs prepared with default values set.
I 00:00:00.157722 executorch:qnn_executor_runner.cpp:506] Model executed successfully.
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/2.31.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.4.0/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[WARNING] [Qnn ExecuTorch]:  <W> Function not called, PrepareLib isn't loaded!

[SS-JIA]

Btw @moxprox, ExecuTorch also has a Discord which has a qnn delegate channel. If you are interested, feel free to join! In most cases it is easier to go back and forth on Discord while troubleshooting.