The 2022–2025 scientific literature on LLM on-device inference validates the majority of both resource/performance and device/hardware terms in the keyword matrix.
tokens/s, tok/s), the latency family (latency, KV cache, context length), the thermal/energy cluster, and the quantization vocabulary each appear in multiple peer-reviewed papers as primary research vocabulary. The matrix's hybrid design — mixing academic benchmarking vocabulary with practitioner jargon — is well-supported. The main gap: TTFT and TPOT/ITL are the canonical latency decomposition metrics in academic benchmarking and are entirely absent from the grid.
The most throughput-specific terms in the resource axis appear with striking consistency across the literature. tokens/s and tok/s are the near-universal decode-throughput metric; PowerInfer-2, for instance, reports exactly "11.68 tokens/s" when running a 47B-parameter LLM on a smartphone (PowerInfer-2, arXiv:2406.06282), and every benchmarking paper in the survey set follows the same convention. latency is universally present but in academic papers it is almost always decomposed into phase-specific sub-metrics (see the gap analysis below). KV cache is discussed in multiple dedicated proposals for on-device compression or offloading, confirming it is both a symptom label and an active area of optimization research (MiniKV, arXiv:2411.18077; mzCache, arXiv:2609.01338). quantization and context length appear as independent variables in virtually every systematic evaluation (arXiv:2505.15030).
The thermal and energy cluster is equally well attested. thermal and throttling/throttled are named as a primary constraint in sustained-inference measurement studies: one paper documents that iPhone 16 Pro throughput falls from 40.35 tok/s to 22.56 tok/s within two inference iterations due to thermal throttling (arXiv:2603.23640). battery appears in mobile deployment papers as a user-facing metric ("20 conversations consume up to 6–25% of battery in under 15 minutes") (arXiv:2609.11940), while energy and power are used in instrumented-measurement studies reporting Joules per token or steady-state Watts (EnerInfer, arXiv:2606.23001; PELM, arXiv:2609.09662). The three are complementary: battery covers end-user perception, energy covers cumulative cost, power covers the instantaneous hardware envelope.
GGUF has completed the journey from practitioner format to academic artifact: 2024–2025 papers call it "a practical standard for portable deployment across ARM CPUs and NVIDIA GPUs" and cite its Q4_K_M sub-variant in methodology sections (arXiv:2604.24785).
tokens/s · tok/s · latency · KV cache · context length · quantization · thermal · throttling · throttled · battery · energy · power · RAM · GGUF all appear as primary research vocabulary in ≥2 peer-reviewed papers from 2022–2025.
The matrix deliberately includes terms drawn from the symptom-seeded vocabulary principle of Islam et al. (2019), and four of those terms — VRAM, memory leak, regression, and slow/slow inference — are not found as primary vocabulary in academic papers about on-device LLM inference.
VRAM is discrete-GPU nomenclature; on-device papers refer instead to "DRAM", "memory footprint", or "peak RSS".memory leak does not appear in formal benchmarking papers; it surfaces in GitHub issue trackers describing growing memory consumption during multi-turn sessions.regression is used in practitioner context for version-to-version performance degradation; papers describe the same phenomenon as "throughput degradation" or "performance drop".slow and slow inference are too colloquial to appear as metric names; papers quantify the same observation with tokens/s or latency values.The literature draws a consistent distinction between hardware architecture labels (which function as primary research subjects) and consumer brand/model names (which appear only incidentally as test-device identifiers).
NPU is validated at the highest level: it appears in conference paper titles at ASPLOS 2025 ("When NPUs Are Not Always Faster") (arXiv:2605.27435; ACM DL 10.1145/3669940.3707239). Snapdragon is named as the concrete SoC under study across benchmark papers covering the Snapdragon 8 Gen 3, 8 Gen 1, and X Elite families (arXiv:2501.14794). Apple Silicon is confirmed as a recognized hardware category analogous to "Snapdragon", appearing in paper titles explicitly benchmarking M1/M2/M4 systems (ACM DL 10.1145/3771563). Jetson and raspberry are validated as representative embedded/IoT edge testbeds across survey papers. ARM is consistently used at the ISA level, naming Cortex-A76 cores and Mali GPUs as specific research subjects.
By contrast, ios, iphone, ipad, and pixel are virtually absent as primary research terms. These terms are still justified in the matrix for GitHub issue mining — end users filing issues do name their device model — but their low coverage in academic papers limits their utility for literature-based validation.
Academic benchmarking papers for on-device LLM inference have converged on exactly two phase-specific metrics:
Both appear in paper titles, abstract result summaries, and benchmark comparison tables across MobiBench, lm-Meter, and the Characterizing Mobile SoC paper (MobiBench, arXiv:2609.13159; lm-Meter, arXiv:2510.06126; arXiv:2501.14794). The Hydra paper is titled "Phase-Aware Workload Characterization of LLM Inference" precisely because the prefill/decode distinction is the dominant structuring principle of on-device performance analysis (arXiv:2608.25053).
Note that "prefill" was added to the grid on 2026-10-10 (approved with 65% precision on llama.cpp). TTFT and TPOT/ITL remain as secondary-axis candidates for a future extension.
The literature identifies memory bandwidth as the binding bottleneck for decode-phase throughput on every class of edge device: "LLM inference is largely memory-bound," with the Snapdragon 8 Gen 3's theoretical peak of 68 GB/s named as the hardware ceiling (arXiv:2501.14794). Memory footprint and peak DRAM are the preferred academic forms of the memory size constraint, more precise than bare RAM.
Three sub-component labels used as independent research subjects are missing from the device axis:
SoC (System-on-Chip) — the primary unit of analysis in characterization papers ("Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference") (arXiv:2501.14794)Hexagon (Qualcomm's NPU/HTP brand) — named as "a representative mobile NPU due to its typical architecture, widespread adoption, and accessible SDK" (arXiv:2509.23324)ANE — documented as present on over two billion devices yet "remains largely unused for LLM workloads"; ANE-targeted inference is an active research problem (arXiv:2603.06728; arXiv:2604.18788)All three would serve as secondary axis terms to improve recall for academic papers without degrading precision. The term heterogeneous (heterogeneous SoC, heterogeneous inference) also appears in multiple titles as a descriptor for combined CPU+GPU+NPU execution.
| Missing term | Axis | Evidence level | Recommendation |
|---|---|---|---|
TTFT | Resource | High — paper titles, abstract summaries | Add to RESOURCE axis (secondary) |
TPOT / ITL | Resource | High — paper titles, benchmark tables | Add to RESOURCE axis (secondary) |
SoC | Device | High — primary unit of analysis in char. papers | Add to DEVICE axis (secondary) |
Hexagon | Device | Medium — named in ≥2 systems papers | Add to DEVICE axis (secondary) |
ANE | Device | Medium — active research problem | Add to DEVICE axis (secondary) |
memory bandwidth | Resource | High — discussed in every hardware char. paper | Consider as secondary axis term |
A wave of empirical software-engineering papers published in 2025 studied bugs and issues in LLM inference engines by mining GitHub repositories — methodology that directly parallels this study.
Manually labels 929 bugs across llama.cpp, MLC-LLM, vLLM, DeepSpeed, and TensorRT-LLM, achieving Cohen's κ = 0.927 on a 6-symptom × 28-root-cause taxonomy (source). Two repositories are shared with this study (llama.cpp, MLC-LLM), making it a direct comparator. Its S4 Abnormal Performance symptom category — covering excessive resource consumption and abnormal token generation latency — maps to the same phenomenon as RQ1. Liu et al. treat it as one of six symptoms (6% frequency), which likely underestimates on-device performance issues because their sample includes cloud-optimized engines where performance issues manifest differently.
Studied 994 closed issues from 15 LLM open-source projects including Ollama; proposed a 15-category taxonomy where Performance Issue (7.14%) and Memory Issue (3.92%) are treated as separate top-level categories (source). The explicit separation of memory from performance as distinct top-level categories is a direct precedent for this study's resource_category field.
The methodological predecessor for on-device mobile inference faults specifically; collects 304 faults from TF Lite and Core ML and proposes a 23-category taxonomy (source). Should be cited as the baseline from which this study advances by covering the post-2022 generation of on-device LLM runtimes.
The standard RQ structure across all related empirical studies converges on: (1) a symptom/category taxonomy, (2) root-cause analysis, and (3) fix-effort characterization. This study's RQ1 (symptom categories) and RQ3 (quantization impact) align with this structure.
The keyword matrix is well-grounded in the 2022–2025 literature. The principal corrections needed are on the academic-retrieval side:
SoC, Hexagon, and ANE represent sub-component granularity at which recent systems papers operate and would increase recall without sacrificing device-specificity.KEYWORD-GRID rather than primary queries, preserving the existing structure while filling in academic coverage.The taxonomy question is answered decisively: Liu et al. (2506.09713) and Cai et al. (2409.16559) together provide a citation-backed foundation for the RQ1 category scheme, and their separation of memory from performance validates this study's choice to do the same.