llama.cpp b9966 makes one change: 29 std::regex patterns that recompiled on every tensor-per-token callback are now static const (release notes).
- The bug:
llama_meta_device_get_split_state()recompiled 29 regex patterns on every call - The impact: In
-smtensor mode, the callback fires once per tensor per token — regex compilation dominated the decode thread in profiling - The fix: Mark patterns
static constso they compile once (PR #24710) - Thread safety: Local statics are thread-safe since C++11; patterns are literal and stateless, so behavior is unchanged
This is the kind of overhead that doesn’t surface in single-GPU benchmarks. It only bites when you’re running multi-GPU tensor splits with a large model — the callback path runs for every tensor allocation check during decode, rebuilding identical regex objects thousands of times per second. Pure waste, zero functional purpose.
NOTE
The fix only affects -sm (tensor split mode). Single-GPU and row-split configurations won’t see a measurable difference — the callback isn’t on the hot path.
Builds available for macOS arm64/x64, iOS, Linux (CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16), Android arm64, and Windows (CPU, CUDA 12.4, CUDA 13.3, Vulkan, OpenVINO, SYCL, OpenCL Adreno).