Skip to content

llama.cpp b9966: static regex fix kills decode bottleneck in tensor-split mode

llama.cpp b9966 fixes a decode-thread bottleneck: 29 std::regex patterns recompiling per tensor per token in -sm mode are now static const. PR #24710.

llama.cpp b9966 makes one change: 29 std::regex patterns that recompiled on every tensor-per-token callback are now static const (release notes).

  • The bug: llama_meta_device_get_split_state() recompiled 29 regex patterns on every call
  • The impact: In -sm tensor mode, the callback fires once per tensor per token — regex compilation dominated the decode thread in profiling
  • The fix: Mark patterns static const so they compile once (PR #24710)
  • Thread safety: Local statics are thread-safe since C++11; patterns are literal and stateless, so behavior is unchanged

This is the kind of overhead that doesn’t surface in single-GPU benchmarks. It only bites when you’re running multi-GPU tensor splits with a large model — the callback path runs for every tensor allocation check during decode, rebuilding identical regex objects thousands of times per second. Pure waste, zero functional purpose.

NOTE

The fix only affects -sm (tensor split mode). Single-GPU and row-split configurations won’t see a measurable difference — the callback isn’t on the hot path.

Builds available for macOS arm64/x64, iOS, Linux (CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16), Android arm64, and Windows (CPU, CUDA 12.4, CUDA 13.3, Vulkan, OpenVINO, SYCL, OpenCL Adreno).