llama.cpp build b10282 adds speculative decoding counters to the server’s /metrics endpoint, with parameter names aligned to match vLLM exactly.
The change (PR #26389) gives operators running llama-server a way to measure whether speculative decoding is actually improving throughput — previously opaque without custom instrumentation.
What shipped in b10282:
- Spec-decode counters at
/metrics— track accepted and rejected draft tokens - vLLM-aligned param names — drop-in compatibility for existing Grafana panels and alerting rules
- CUDA 13.3 DLLs for Windows x64 — listed alongside the existing CUDA 12.4 artifact line
- KleidiAI on macOS arm64 — present in the build matrix but marked DISABLED
Speculative decoding lets a small draft model propose tokens that a larger model verifies in parallel. When the draft guesses well, you get multiple tokens per forward pass. When it guesses poorly, you pay the verification cost with no benefit. The new counters make that acceptance ratio visible at the Prometheus scrape level.
The vLLM naming alignment matters if you run mixed inference stacks. A dashboard built for vLLM’s spec-decode metrics should work against llama.cpp without relabeling.
NOTE
macOS arm64 with KleidiAI is listed in the build matrix but marked DISABLED for b10282. If you were waiting on KleidiAI acceleration, this build won’t ship it.
The full artifact matrix covers 27 platform targets, including Ubuntu s390x, Windows arm64 OpenCL Adreno, SYCL FP16, and openEuler on Huawei Ascend 910b with ACL Graph.
Available now as prebuilt binaries on the b10282 release page. Pull the artifact for your platform and scrape /metrics after enabling spec-decode.