Skip to content

llama.cpp b9968: int8 MoE prefill optimization for Adreno GPUs

llama.cpp b9968 adds int8 dp4 dense and MoE GEMM OpenCL kernels for Qualcomm Adreno GPUs. Build artifacts span 20+ platform configs including Windows arm64 OpenCL.

llama.cpp build b9968 ships int8 dot-product-4 dense and MoE GEMM kernels for OpenCL, targeting Qualcomm Adreno GPUs. PR #25537 was co-authored by Li He from Qualcomm’s QTI team.

What changed:

  • int8 dp4 dense GEMM — new OpenCL kernels for int8 quantized matrix multiply using dot-product-4 instructions
  • MoE prefill optimization — Mixture-of-Experts models get faster prefill on Adreno GPUs
  • OpenCL refactor — backend code restructured alongside the new kernels
  • Windows arm64 (OpenCL Adreno) artifact — prebuilt binary for Snapdragon X-class laptops with Adreno GPUs

Adreno is the GPU in Qualcomm Snapdragon SoCs — most Android phones and Windows-on-ARM devices. int8 dp4 uses hardware dot-product instructions to accelerate quantized inference, which matters most for MoE models where prefill latency is dominated by expert GEMM operations.

Build artifacts span 20+ configs: macOS arm64/x64, iOS XCFramework, Ubuntu CPU/Vulkan/ROCm 7.2/OpenVINO/SYCL, Windows CPU/CUDA 12.4/CUDA 13.3/Vulkan/SYCL/HIP, and openEuler for Ascend NPU (310p, 910b with ACL Graph).

NOTE

macOS Apple Silicon with KleediAI is marked DISABLED in this build’s artifact matrix. openEuler non-Ascend builds are also disabled.

Available now as prebuilt binaries on the b9968 release page. Android arm64 CPU and Windows arm64 OpenCL Adreno artifacts are included for testing the new kernels on-device.