Single source of truth for the GitHub Search API query strategy. Every term has a literature reference. Every rejection is recorded.
We follow Islam et al.'s (2019) search-vocabulary construction protocol, adapted from Deep Learning bugs to on-device inference execution cost:
"out of memory", OOM, slow, thermal throttling, not "allocator memory pressure".latency/ms, energy/battery, RAM/VRAM, tok/s.tok/s (not just "throughput"), KV cache, GGUF, Q4_K_M.Exclusion rule: a term is rejected if, in a 20-result pre-test, more than 50% are build/compilation errors or conceptual questions. Rejections are recorded in §5 and §5.1.
Symptoms and execution-cost metrics. These terms determine what the problem is.
| Term | Category | Technical Justification | Literature Reference |
|---|---|---|---|
"out of memory" |
Resource | Most frequent verbal form of the memory-exhaustion symptom; appears in free text before any diagnosis. Leading symptom of the crash category in DL bugs. | Islam et al. 2019; Islam et al. 2020 |
OOM |
Resource | Dominant acronym in Linux/Android kernel logs (oom_reaper, lowmemorykiller); captures reports that paste the log. |
Islam et al. 2019; Codabux et al. 2025 |
"memory leak" |
Resource | Distinguishes progressive exhaustion (multi-turn session) from exhaustion at load time. Separates static from dynamic cost. | Islam et al. 2020; Dantas Cruz et al. 2020 |
"KV cache" |
Resource | The only memory component that grows linearly with context; primary error term for any estimator based on weight-file size alone. Direct target of RQ3. | Imai et al. 2024; Cai et al. 2026 |
"context length" |
Resource | Configuration parameter cited when reporting OOM ("works at 2k, breaks at 8k"); provides quasi-experimental evidence at no annotation cost. | Imai et al. 2024; Codabux et al. 2025 |
RAM |
Resource | Unit in which mobile users reason about feasibility ("8 GB RAM doesn't run it"); recovers issues without a stack trace. | Tu et al. 2026 |
VRAM |
Resource | Equivalent for GPU backends (ROCm, Metal, Vulkan); needed to cover Apple Silicon and dGPU cohort. Note: academic papers use DRAM/RSS; this term is GitHub-issue-calibrated. | Cai et al. 2026; Imai et al. 2024 |
"tokens/s" |
Resource | Canonical decode-throughput metric; the spelled-out form used in benchmark tables pasted inside issues. | Imai et al. 2024; Dantas Cruz et al. 2020 |
"tok/s" |
Resource | Overwhelmingly dominant abbreviation in llama.cpp/Ollama. Omitting it caused 100% recall loss in pre-tests on ggml-org/llama.cpp. |
Codabux et al. 2025 |
latency |
Resource | Academic/tooling term for response time; recovers TTFT reports that do not mention throughput. | Imai et al. 2024; Tu et al. 2026 |
slow |
Resource | Lay term for latency. Low precision in isolation; high recall of end-user reports. Used only in conjunction with a device term. | Islam et al. 2019 |
regression |
Resource | Isolates version-to-version degradations — the strongest causal evidence in an observational study (same hardware, same model, different build). | Islam et al. 2020; Thomas et al. 2024 |
thermal |
Resource | Names the regime where performance stops being a function of the model and becomes a function of the thermal envelope — a structural failure for any static predictor. | Tummalapalli et al. 2026 |
throttling |
Resource | Nominal form; the most common inflection in issue text. Replaces the defunct stem throttl (GitHub Search is whole-word only). |
Tummalapalli et al. 2026; Cai et al. 2026 |
throttled |
Resource | Past-participle form ("device got throttled"); complements throttling for reports describing the outcome rather than the process. |
Tummalapalli et al. 2026 |
battery |
Resource | Cost perceived by the mobile user; often the only metric reported when no profiling tool is available. | Zou et al. 2026 (EnerInfer) |
energy |
Resource | Instrumented-measurement term (J, mWh); recovers issues with serious measurement. Stargazer v1 regression target. | Zou et al. 2026; Dantas Cruz et al. 2020 |
power |
Resource | Instantaneous power (W); needed for Jetson/SoC issues where the limit is the power envelope, not battery charge. | Zou et al. 2026; Tummalapalli et al. 2026 |
quantization |
Resource | Central independent variable of RQ3: reporter almost always names the scheme (Q4_K_M, int4, fp16) when comparing costs. | Cai et al. 2026; Tu et al. 2026 |
"inference speed" / "slow inference" |
Resource | Frequent fixed collocations in issue titles; recover reports using neither latency nor tok/s. |
Codabux et al. 2025; Imai et al. 2024 |
"prefill" NEW |
Resource | LLM inference phase preceding token generation; source of distinct bottlenecks from decode. Reporters name the phase when describing regressions (prefill slow, prefill OOM). Pre-test: 13/20 = 65% scoped to ggml-org/llama.cpp. Global test (9/20) depressed by server-side repos not in corpus — grid queries always run repo-scoped, so 65% is the operative precision. APPROVED. |
Imai et al. 2024; Liu et al. arXiv:2506.09713; Cai et al. 2026 |
Execution environment. These terms determine where the problem occurs, separating the on-device cohort from server-side noise.
| Term | Category | Technical Justification | Literature Reference |
|---|---|---|---|
android |
Device | Largest on-device platform by volume; where memory and thermal constraints are simultaneously binding. | Tummalapalli et al. 2026; Cai et al. 2026 |
ios / iphone / ipad |
Device | Cohort with a per-process memory limit enforced by the OS (jetsam), producing a failure mode qualitatively distinct from Linux OOM. | Cai et al. 2026 |
pixel |
Device | Names a concrete device with a Tensor TPU; allows an issue to be tied to an identifiable SoC (RQ2 requirement). | Cai et al. 2026 |
snapdragon |
Device | Dominant SoC family on Android; the reporter names the chip more often than the device model. | Cai et al. 2026; Tummalapalli et al. 2026 |
NPU |
Device | Accelerator whose cost behavior diverges most from FLOPs-based predictions; explicit target of the hidden-bottleneck literature. | Cai et al. 2026 ("Is Your NPU Ready for LLMs?") |
"on-device" |
Context | Term the community itself uses for self-identification (executorch, LiteRT); high precision value. | Codabux et al. 2025 |
mobile |
Context | Generalization that recovers issues without a named platform. | Tummalapalli et al. 2026 |
edge |
Context | Same role for SBCs/embedded devices; connects directly to Stargazer v1's framing (Raspberry Pi cluster). | Dantas Cruz et al. 2020; Tu et al. 2026 |
"Apple Silicon" |
Device | Unified-memory cohort, where RAM and VRAM collapse into one variable — an important edge case for the cost model. | Imai et al. 2024 |
Jetson |
Device | Edge device with discrete GPU and configurable power envelope (nvpmodel); supplies controlled-power reports. |
Zou et al. 2026; Tu et al. 2026 |
ARM |
Context | ISA shared by the on-device cohort; recovers kernel/backend issues (NEON, SVE, aarch64). | Tu et al. 2026 |
raspberry |
Device | Methodological continuity with Stargazer v1's hardware. | Dantas Cruz et al. 2020 |
GGUF |
Context | Artifact format that carries the quantization scheme in the file name; makes RQ3 extractable from issue text. 2025 papers confirm it is "a practical standard for portable deployment across ARM CPUs and NVIDIA GPUs". | Codabux et al. 2025 (native jargon); arXiv:2604.24785 |
INT4 NEW |
Device/Context | Dominant quantization precision for on-device LLM deployment; appears in issue text as model descriptor ("INT4 model crashes"). Complements quantization with the specific precision value reporters include when comparing costs. Pre-test with device terms: 9–12/20 = 45–64%. APPROVED. |
MLSys 2024 Best Paper (AWQ/INT4 deployment); arXiv:2601.14277; Tu et al. 2026 |
Added following Issue #5 and the MLOps/LLMOps literature search of 2026-10-10. Treat the training-to-production transition as a complementary, inclusive axis, not a restrictive filter. Six variants were tested with the 20-result pre-test; five failed (recorded in §5.1) and one was approved:
| Term | Category | Justification | Status |
|---|---|---|---|
"model conversion" AND (GGUF OR ollama OR NPU) |
Deployment/Transition | Covers conversion to on-device deployment formats (ONNX/TFLite/GGUF/NPU), where a large share of on-device cost bugs originate. Pre-test: 12/20 = 60% (full-text read). APPROVED. | QUERY-4 |
"model export" AND (GGUF OR executorch OR "mlc-llm") |
Deployment/Transition | Pre-test: 5/20 = 25%. Noise from documentation issues and repos outside LLM inference cost scope. Term captures the export action in general, not the cost symptom. | REJECTED |
The remaining MLOps vocabulary (deployment stage labels) should enter as a manual annotation field (stage ∈ {development, deployment, production-operation}) during the 03_annotation step, not as free-text search keywords.
| Constraint | Value | Project implication |
|---|---|---|
| Query length | ≤ 256 characters (not counting operators/qualifiers) | Not the bottleneck: our queries run 71–90 chars |
| Boolean operators | at most 5 AND/OR/NOT per query | The real bottleneck. Limits each query to ~6 boolean terms |
| Results per query | 1,000 (100 × 10-page pagination) | Requires repo: decomposition to avoid truncation |
| Rate limit (authenticated) | 30 req/min on /search/* | Mandatory backoff in mine_issues.py |
QUERY-1 (90 total chars / 65 countable chars / 5 operators):
is:issue ("out of memory" OR OOM OR "KV cache") AND (android OR snapdragon OR "on-device")
Axis: Memory pressure × mobile cohort
Covers: "out of memory", OOM, "KV cache", android, snapdragon, "on-device"
RQ addressed: RQ1 (Memory category), RQ2
QUERY-2 (82 total chars / 57 countable chars / 5 operators):
is:issue ("tokens/s" OR "tok/s" OR latency) AND ("Apple Silicon" OR ios OR jetson)
Axis: Throughput and latency × Apple / GPU-edge cohort
Covers: "tokens/s", "tok/s", latency, "Apple Silicon", ios, jetson
RQ addressed: RQ1 (Latency and Throughput categories), RQ2
QUERY-3 (71 total chars / 46 countable chars / 5 operators): is:issue (thermal OR throttling OR battery) AND (npu OR mobile OR edge) Axis: Thermal and energy envelope × NPU / mobile / embedded cohort Covers: thermal, throttling, battery, npu, mobile, edge RQ addressed: RQ1 (Energy and Temperature categories), RQ3
QUERY-4 (68 total chars / 3 operators) — Deployment axis: is:issue "model conversion" AND (GGUF OR ollama OR NPU) Axis: Deployment/transition × on-device conversion ecosystem Covers: "model conversion", GGUF, ollama, NPU RQ addressed: RQ1 (Toolchain/Artifact entry point), RQ3
The miner appends repo:<owner>/<name> to each query. Qualifiers do not count toward the 256-char limit or the operator cap.
KEYWORD-GRID)Cartesian product of resource term × device term; one query per pair (0–2 operators). Parsed programmatically by mine_issues.py via parse_grid_terms(). Total: 22 × 16 = 352 queries per repository.
NEW 2026-10-10 "prefill" added to RESOURCE; INT4 added to DEVICE.
| Rejected term | Reason for rejection |
|---|---|
crash | Pre-test: 17/20 were build failures/segfaults with no resource metric. Too ambiguous a symptom for the cost axis. |
performance | Pre-test: 14/20 were feature requests ("performance roadmap"), not measured-cost reports. |
CUDA | Anchors results in datacenter/desktop-dGPU — exactly the cohort the RQs exclude. |
bug | Orthogonal to the cost axis; does not discriminate. |
temperature | Collides with the LLM sampling hyperparameter temperature. Replaced by thermal + throttling/throttled. |
Six variants tested for the complementary axis (Issue #5, 2026-10-05). Rejected if noise > 50% in a 20-result pre-test.
| Term | Variant tested | Related/20 | Reason |
|---|---|---|---|
deployment | free text | 0/20 | Dominated by generic "Deployment" titles from unrelated projects |
deployment | + OOM / tok/s / latency | 6/20 | Datacenter-scale LLM serving (vLLM, sglang) — excluded cohort |
deployment | + resource + device (mobile, edge, on-device) | 3/20 | Got worse; "edge" collided with NVIDIA product name |
deployment | scoped to repo:ggml-org/llama.cpp | 7/20 | Best variant; still below the 50% cutoff |
serving | free text | 0/20 | Food-serving homonym (recipes, restaurant apps) |
"inference server" | free text, exact phrase | 0/20 | All results were architecture/feature tasks, never cost symptoms |
production | free text | 0/20 | CI/CD, manufacturing, blockchain, bots |
rollout | free text | 0/20 | RL training-episode homonym within the ML domain |
"model export" | AND (GGUF OR executorch OR "mlc-llm") | 5/20 | Documentation issues, feature requests, unrelated repos |
"unsupported op" | AND (NPU OR executorch OR "mlc-llm") | 4/20 | LuaJIT optimizer, analytics DB, Discord bot — no LLM inference connection |
Common pattern: process-stage vocabulary (deployment, serving, production) and mechanism-naming terms ("model export", "unsupported op") fail because issue reporters write the symptom, not the stage or internal mechanism. This empirically confirms Islam et al.'s principle for the MLOps axis specifically.
| Research Question | Terms that instrument it | Output column |
|---|---|---|
| RQ1 — Which categories of resource problems are reported when running LLMs on-device? | out of memory, OOM, memory leak, RAM, VRAM, latency, slow, tok/s, tokens/s, thermal, throttling, throttled, battery, energy, power, "prefill" |
resource_category ∈ {Memory, Latency, Energy, Temperature, Throughput, Other} |
| RQ2 — Which model × hardware × backend combinations produce the most problems? | android, ios, pixel, snapdragon, NPU, Apple Silicon, Jetson, ARM, raspberry × toolchain names |
(model, hardware, repo) triple from filter_issues.py |
| RQ3 — What is the impact of quantization parameters on reported execution cost? | quantization, GGUF, INT4, KV cache, context length, "model conversion" AND (GGUF OR ollama OR NPU) |
quantization + numeric_metrics |
| Entry point | Definition | Indicator terms |
|---|---|---|
| Artifact | Cost from a property of the downloaded model | quantization, GGUF, KV cache, context length |
| Toolchain | Cost from the runtime/compiler/backend, not the model | regression, toolchain names, memory leak |
| Platform | Cost from the device and its physical envelope | thermal, throttling, battery, power, NPU, snapdragon |
See the full bibliography page for annotated entries. Key matrix references: