Skip to content

llama.cpp b9931: OpenCL MoE prefill skips padding tiles to cut wasted compute

llama.cpp build b9931 adds two OpenCL optimizations for MoE prefill GEMM that skip padding tiles, reducing wasted compute with byte-identical output.

llama.cpp build b9931 ships two OpenCL optimizations that cut wasted compute in Mixture-of-Experts prefill — by skipping padding tiles entirely.

The changes target a specific inefficiency: MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles, but at low tokens-per-expert most of those tiles are padding. The release adds:

  • Ragged-tile FP16 skip — when a tile’s upper 16 slots are all padding (router index 0xFFFFFFFF), the kernel skips the second dotx16_reduce8 half. Applied to all eight *_f32_ns MoE GEMMs. Default on; opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
  • Quarter-granularity tile-skip — replaces two dotx16_reduce8 calls with four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group independently. Controlled by GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32}.

Both paths are numerically and byte-identical to the non-skipped version — skipped lanes are padding, so no output changes. The win is pure compute reduction on OpenCL backends during prefill for MoE models like DeepSeek-V3 or Mixtral, where token distribution across experts is sparse and tiles fill unevenly.

NOTE

These optimizations are OpenCL-only. CUDA and Metal backends have their own MoE kernels and are unaffected by these env vars.

Available in llama.cpp b9931 — no migration needed, FP16 skip is on by default.