The 2022–2025 literature on on-device and edge LLM inference supplies a well-defined vocabulary that both identifies edge-specific issues and actively excludes cloud MLOps noise.
GGUF, Q4_K_M, Q8_0model conversionbackend loweringONNX exportoperator fallbackgraph breakthermal throttlingbattery drainmemory pressureOTA model updatedevice heterogeneityDVFSprefill / prefill latencyKV cache saturationsession rotationdecode latencymodel offloadingcontext degradationThe clearest separator between edge and cloud MLOps vocabularies is a set of terms that simply do not appear in cloud serving infrastructure.
GGUF is the most powerful single discriminator: it is the standard quantized model container for CPU/mixed CPU-GPU inference in the llama.cpp and Ollama ecosystems, and a 2025 paper confirms it has become "a practical standard for portable deployment across ARM CPUs and NVIDIA GPUs" (arXiv:2604.24785). A GitHub issue containing GGUF is, by construction, not about cloud serving. The sub-format variants Q4_K_M, Q8_0, and Q4_0 carry the same discrimination at higher specificity; quality degrades "gradually from Q8_0 to Q4_K_M (0.51% avg loss), then precipitously from Q4_K_M to Q2_K (7.94%)" (arXiv:2601.14277).
OTA model update and differential update are equally unambiguous: they describe wirelessly distributing new model weights to deployed devices, a problem that does not exist in cloud serving where models are simply re-deployed server-side. TinyMLOps is the named subfield label for MCU-class operations. SLM (Small Language Model) is the class label for LLMs sized for edge/mobile deployment (arXiv:2410.20011).
auto-scaling · Kubernetes · KServe · serverless inference · model registry (in the centralized MLflow/SageMaker sense) — any GitHub issue using these terms without an explicit edge qualifier is cloud infrastructure and should be excluded from corpus construction.
The quantization format space has a clear two-tier structure that matters for keyword design.
Deployment formats — GGUF variants, AWQ, and generic INT4/INT8 — appear as primary vocabulary in systems papers and practitioner deployment documentation. Training/adaptation formats — NF4 via bitsandbytes — appear in fine-tuning papers but practitioners consistently recommend converting away from NF4 before deployment, making it a training vocabulary term that would be misleading in an edge inference issue query.
AWQ (Activation-aware Weight Quantization) won the MLSys 2024 Best Paper Award specifically for on-device LLM compression, enabling a "3–4× performance boost vs. FP16 via the TinyChat framework on edge platforms" (MLSys 2024). By 2024, major model releases began shipping AWQ checkpoints alongside standard weights, making it both a research algorithm and a distribution format choice.
GPTQ appears across nearly all edge deployment papers from 2022 onward as a Hessian-based layer-wise quantization baseline, though it is primarily suited to NVIDIA GPU serving and therefore less discriminative of true on-device edge contexts (arXiv:2306.00978).
quantization error · accuracy degradation · fidelity loss · distributional drift (arXiv:2609.07664) · unsupported op / operator fallback · backend compilation failure · weight layout mismatch · GGUF conversion error. Note: "unsupported op" was rejected as a standalone search term (4/20 = 20% precision in the pre-test) but remains valid as an annotation label.
Cloud MLOps treats model deployment as a packaging and registry operation. On-device MLOps introduces a directed conversion pipeline with distinct stages and vocabulary at each stage — vocabulary that maps directly to the categories of GitHub issues filed against executorch, mlc-llm, and llama.cpp.
The canonical pipeline: export → intermediate format → quantization → compilation/lowering → packaging into runtime artifact.
torch.export is ExecuTorch's "execution graph capture mechanism" that converts PyTorch modules into static AOT graphs, introducing an "Export IR" and "Edge Dialect" restricted to fewer than 300 Core ATen operators (arXiv:2605.08195). ONNX export is the standard intermediate step for multi-backend pipelines, and PyTorch → ONNX → TFLite is an established "multi-step pipeline for on-device inference" (arXiv:2511.13453). Failure vocabulary: dynamic shapes failure, data-dependent control flow, and custom operator support issues.
ExecuTorch's "backend lowering" delegates subgraphs to hardware accelerators via a "delegate blob" (backend-specific compiled representation); MLC-LLM uses mlc_llm compile to generate "a model library (a shared library containing the compiled executable code)". The term model conversion is positioned as a primary research term covering "operator compatibility handling, runtime integration, and precision verification" (arXiv:2509.23324). This is why "model conversion" AND (GGUF OR ollama OR NPU) was approved as QUERY-4 in the keyword matrix.
The three named phases of on-device inference execution are: engine initialization, prefill (context processing/KV cache construction), and decode (token generation).
The operational challenges that make edge MLOps distinct from cloud MLOps generate a vocabulary cluster that is essentially impossible to confuse with cloud issues.
Thermal vocabulary has particularly strong 2024–2025 literature support. One measurement study documents that iPhone 16 Pro throughput falls from 40.35 tok/s to 22.56 tok/s within two inference iterations due to thermal throttling (arXiv:2603.23640). DVFS (Dynamic Voltage and Frequency Scaling) is the hardware mechanism that translates thermal/power events into compute degradation (arXiv:2512.01644).
Energy vocabulary: energy per token / joules per token is the on-device analog of cloud's "cost per token" and is the canonical efficiency metric in multiple 2025 papers. Battery drain is documented as "sustained inference session potentially draining 1% battery per minute" in mobile deployment papers.
OTA and fleet vocabulary: version fragmentation (devices in a deployed fleet running different model versions due to asynchronous update schedules) and client-side fallback / model rollback at scale are named operational concerns for edge fleets. Differential model update is documented as achieving 20–50× bandwidth reduction vs. full model redownload.
| Register | Memory terms | Latency terms | Throughput terms |
|---|---|---|---|
| Academic papers | DRAM, memory footprint, peak RSS | TTFT, TPOT, ITL, TBT | tokens/s, tok/s |
| GitHub issues | RAM, VRAM, memory leak | latency, slow, "not fast enough" | tokens/s, tok/s (shared) |
| Matrix coverage | Both registers ✓ | Issue register only (TTFT/TPOT missing) | Both registers ✓ |
Terms ambiguous across edge and cloud that require qualifiers: LoRA and TTFT appear in both cloud and edge contexts — they are not discriminative alone and need pairing with edge qualifiers (on-device LoRA, TTFT on Pixel). Phase-level and kernel-level granularity terms (phase-level latency, kernel-level bottleneck, powersave governor) make them discriminative in combination.
The on-device LLMOps vocabulary divides into three tiers with distinct roles in keyword matrix design:
model conversion, backend lowering, operator fallback, graph break — because they are structurally impossible to confuse with cloud serving.thermal throttling, energy per token, OTA model update, differential update, device heterogeneity — equally unambiguous, but serve a different filtering role: the operational management layer.memory bandwidth, SoC, Hexagon, ANE — increase recall against academic benchmarking papers without trading away device-specificity.The practical recommendation: treat the three clusters as complementary rather than competing. Pipeline conversion terms anchor the format/compatibility query group; hardware constraint terms anchor the performance/monitoring query group; academic metric terms anchor cross-referencing with the growing body of 2023–2025 on-device LLM benchmarking papers.