Preliminary version · Round 1. The numbers in this document (Kappa, filter precision, and category frequencies) come from a pilot labelling round still under review. They will be recalculated in the definitive round; consult the final version before citing.
Stargazer v2 · Mining Software Repositories
What 7,889 GitHub issues say about on-device inference cost
Empirical evidence that practitioners cannot predict the execution cost of an on-device LLM, and of how much they miss by when they try.
Corpus: 7,889 issues · 11 repositoriesPeriod: 2017-12 to 2026-09Collection: 2026-09-30
01 · Motivation
The v2 experiments measure; the MSR shows what to measure
The six prior experimental tracks established that inference cost is measurable and, in the notebook, predictable. What they do not answer is which cost problems people actually encounter in the field, nor by what magnitude they miss when estimating them.
What the experiment gives
Controlled measurement of three models on two devices. High precision, narrow coverage.
What the MSR gives
Broad coverage of models, devices, and backends, with low precision per report. Defines the scenario space the predictor needs to get right.
Protocol adapted from Islam et al. (ESEC/FSE 2019), who built a taxonomy of deep learning bugs from 2,716 Stack Overflow posts and 500 GitHub commits. Here the object shifts from defect to execution cost.
02 · The claim
It's not that people suffer from cost. It's that they can't predict it.
Weak claim
"There are many issues about cost, so a prediction tool is justified."
Fragile: an issue tracker records anomalies, not the distribution of pain. And most of these issues would not have been avoided by a predictor.
Claim supported by the data
"Practitioners, and production tools themselves, systematically miss when estimating cost from the artifact."
Every issue that supports this claim carries an expected and an observed value, that is, it is already a test case.
The study's inclusion criterion follows from this choice: does the issue record a cost expectation that was violated, with a number?, not merely "the issue talks about cost".
03 · Sources
Eleven repositories, chosen by ecosystem knowledge
The eleven repositories were included by ecosystem knowledge: runtime, compiler, server, vendor SDK, end-user bug tracker. Screening by the selection protocol (criteria M1–M3) has not yet been run; so they cannot be described as selected by it.
Repository
Role
Raw
Retained
Rate
ggml-org/llama.cpp
C/C++ runtime, GGUF format
3,863
330
8.5%
ollama/ollama
server on top of llama.cpp
1,747
124
7.1%
mlc-ai/mlc-llm
TVM compilation for mobile
434
44
10.1%
pytorch/executorch
PyTorch's on-device runtime
433
18
4.2%
microsoft/onnxruntime
multi-backend runtime + NPU
376
28
7.4%
google-ai-edge/mediapipe
on-device pipelines
322
5
1.6%
lmstudio-ai/…-bug-tracker
end-user reports
297
35
11.8%
ggml-org/whisper.cpp
ASR cohort
134
5
3.7%
ml-explore/mlx
native Apple framework
108
20
18.5%
google-ai-edge/LiteRT
TFLite's successor
102
9
8.8%
apple/coremltools
conversion to ANE/CoreML
73
2
2.7%
TOTAL
n/a
7,889
620
7.9%
ROCm/ROCm was in the plan and dropped out: all 300 queries returned HTTP 422 ("the resources do not exist or you do not have permission"). It is 11 repositories, not 12. The retention gap between mlx (18.5%) and mediapipe (1.6%) is a matter of reporting culture, not software quality.
04 · Search strategy
No term goes in without a reference backing it
The traceable matrix has one row per term, with technical justification and a literature reference. Four sources of derivation:
1Symptoms, not causes: the reporter describes what they saw: out of memory, slow, thermal throttl
2Metrics of the object of study: every target the model predicts needs a term: latency, energy, VRAM, tok/s
3Community-native jargon: tok/s (not just "throughput"), KV cache, GGUF, Q4_K_M
4Precision filter by context: every resource term crossed with a device term
The matrix also records what was rejected and why: crash (17 of 20 in the pre-test were build errors), performance (14 of 20 were feature requests), CUDA (anchors in datacenter), temperature (collides with the sampling hyperparameter).
05 · Project constraint
The bottleneck isn't query length: it's the operators
Limits verified against the official REST documentation on 2026-09-29, and how each one shaped the design:
Constraint
Value
Consequence
Query length
≤ 256 chars
not the bottleneck: 46–65 chars in practice
Boolean operators
max 5
is the bottleneck: ~6 terms per query
Results per query
1,000
requires decomposing by repo: and by date
Authenticated rate limit
30 req/min
mandatory backoff; 2 h 44 min of collection
Consequence recorded, not hidden: the matrix's 35 keywords don't fit in 3 queries under the 5-operator cap. The study runs two sets and deduplicates: 3 high-precision primary queries (the ones reported in the paper) and a grid of 20 resource terms × 15 device terms = 300 atomic queries per repository.
06 · Search strategy
The three primary queries
Query
Axis
Chars
Ops
is:issue ("out of memory" OR OOM OR "KV cache") AND (android OR snapdragon OR "on-device")
memory × mobile
65
5
is:issue ("tokens/s" OR "tok/s" OR latency) AND ("Apple Silicon" OR ios OR jetson)
speed × Apple/edge
57
5
is:issue (thermal OR throttling OR battery) AND (npu OR mobile OR edge)
thermal/energy × NPU
46
5
3,636queries executed
3,841API requests
105rate-limit waits
2h50total collection time
Every query is validated against both limits before being sent. Five queries exceeded the 1,000-result cap (all llama.cpp × GGUF) and were redone with recursive bisection by date, recovering +278 issues, 89% of them from 2023–2024.
The funnel documents are generated by script from the data, never written by hand. This is what keeps the paper's number from diverging from the dataset's number.
08 · Method correction
The first filter had 17% precision. We found out why.
The initial criterion required model AND hardware AND numeric metric, searched anywhere in the text. The Eval bug template in llama.cpp makes the reporter paste the entire startup output, and 38% of this repository's raw issues carry that signature. The log alone satisfied all three gates.
Filter v1
Filter v2
Criterion
model AND hardware AND metric, in the whole text
violated cost expectation, in the prose
Issues retained
1,686 (21.4%)
620 (7.9%)
Resource symptom in title
16.5%
49.1%
Estimated precision
~17%
~40%
Hard exclusions
3
1
The two versions are not nested: v2 keeps 339 of v1's retained issues, discards 1,347, and recovers 281 that v1 had rejected. v1 is frozen in baseline_filter_v1/ and is reproducible with --criterion v1; the comparison is auditable, not an assertion.
09 · RQ1
Memory is the cause; speed is the symptom
Category
Primary
%
Secondary
Memory
396
63.9%
n/a
Latency
76
12.3%
205
Other
75
12.1%
n/a
Throughput
67
10.8%
111
Temperature
6
1.0%
9
Energy
0
0.0%
6
Latency and throughput appear 316 times as a secondary category of the same issues in which memory is primary. The recurring pattern: a memory estimation error pushes part of the model onto the CPU, and the user perceives this as slowness.
Design implication. A predictor that gets memory right solves most of the problem, because observed latency is frequently a function derived from where the model fit.
Negative result. Zero issues with Energy as the primary category. The keywords battery, energy, power, mAh, thermal, and throttl were all in the matrix and were run against the 11 repositories. The energy axis is not reported in these projects.
10 · The central finding
The median surprise is a factor of 4
81 issues (13%) report an explicit degradation magnitude. This is the corpus's most robust descriptive, because it does not depend on classification, it comes straight from the reporter's text.
4.0×median
9.0×75th percentile
50×90th percentile
400×maximum reported
Changes the product's accuracy bar. With surprises of this magnitude, a predictor with 20–30% error already changes decisions. The target is not fine numeric precision, it's getting right which side of the "fits / doesn't fit" boundary the configuration is on.
Distribution of reported ratios (112 mentions)
1.1–2×10
2–3×25
3–5×27
5–10×23
10–100×22
> 100×5
Percentage degradations (36 mentions): median 33% · p90 62%
up to 10×above 10×
11 · RQ2
Model × hardware does not concentrate problems
RQ2 asks which model × hardware combinations generate the most problems. Testing with lift, the observed frequency of the pair over the expected frequency if model and platform were independent:
Pair
n
Lift
glm × Apple Silicon
11
1.54
mistral × Apple Silicon
14
1.23
gemma × Apple Silicon
37
1.13
qwen × Apple Silicon
90
1.06
llama × Apple Silicon
111
0.81
llama × iOS
18
0.54
All lifts between 0.54 and 1.54, indistinguishable from independence. The "hot" pairs are just the product of the marginal frequencies: llama appears in 421 of the 620 and apple-silicon in 201. RQ2, as written, has a negative answer in this corpus. Reporting this is more valuable than forcing a ranking the test doesn't support.
12 · RQ2 reformulated
The variance is in the backend, not the model
Redoing the same analysis with the backend axis, the picture changes entirely. Lifts of 31 and 24.8 against ~1.0 for the model × hardware axis.
Pair
n
Lift
QNN/Hexagon × NPU Hexagon
20
31.0
OpenCL × NPU Hexagon
12
24.8
WebGPU/wasm × iPhone
29
13.9
WebGPU/wasm × iOS
31
11.2
WebGPU/wasm × Android
32
7.5
Vulkan × iPhone
30
3.8
Backend distribution · 74% of issues name at least one
CUDA200
Vulkan134
Metal116
ROCm/HIP96
MLX67
WebGPU/wasm35
QNN/Hexagon20
XNNPACK15
Memory dominates within all backends (49–87%). The only exception is XNNPACK (latency, 47%), which is precisely the mobile-CPU backend, where there is no cross-device memory transfer to get wrong.
13 · RQ3
The problem isn't quantizing the model: it's quantizing the cache
66%of retained issues cite a quantization scheme
40%cite KV cache or context size
90%of those are in the Memory category
52%of Artifact issues cite KV cache
Most-cited schemes: q8_0 (127), q4_k_m (122), f16 (81), q4_0 (64), bf16 (39), fp16 (36). q8_0 leads, and appears more as a KV cache type than a weights type.
The highest-magnitude issues involving quantization are not about quantizing the model, they are about quantizing the cache. llama.cpp#27109: switching the K cache from q8_0 to q4_1 drops prefill from 1,276 to 34 t/s. #27761: on ROCm, types other than q8_0/q4_0 fragment the graph from 2 to 34 splits and drop pp512 by 68%, and the same build on Vulkan doesn't have the problem.
The KV cache is the component that grows with context and that no estimate based on the weights file captures. This is the empirical justification for including n_ctx as an input feature of the predictor.
14 · Predictor scope
Labelling flipped the heuristic's estimate
Entry point
Heuristic (620)
Labelled (27)
Artifact, model, quantization, context
69.2%
25.9%
Toolchain, runtime, backend, build
23.7%
74.1%
Platform, device's physical envelope
7.1%
0.0%
The heuristic flagged Artifact whenever the issue mentioned quantization or KV cache, and 66% do, almost always as context for the report, not as the cause of the cost.
The predictor's reach is smaller than the heuristic suggested. "How many of these issues would Stargazer v2 have prevented?", about 26% of the confirmed ones, not 69%. With n = 27 the interval is wide (≈ 11–46%), but the reversal relative to 69% sits well outside it.
The dominant slice is Toolchain (74%): regressions between versions, backend choice, and scheduler decisions. No static predictor anticipates this. Declaring the scope is methodological strength, and it was the labelling that revealed it.
22 issues satisfy the most demanding criteria simultaneously: explicit magnitude and Artifact and named platform and quantization cited. These are the core of the evaluation benchmark.
15 · Evidence
The three issues that support the paper
pytorch/executorch#22016 · score 20 · iPhone 16 · MLX"whisper-small bf16 uses ~6.7× its file size in RSS and is jetsam-killed, while every sibling stays under 2×"The reporter measured RSS in a freshly launched process for six Whisper variants, with a stable baseline of 554–556 MB. tiny/bf16 uses 1.66× the file; small/bf16 uses 6.7× and dies in jetsam at 3.80 GB. This is the controlled refutation of the "memory ≈ weights size" heuristic, same family, same runtime, same device, varying only size and precision.
ollama/ollama#17099 · score 13 · 24 GB VRAM"0.31.2 regression: memory estimate inflated ~1.2 GiB → no longer fits in VRAM → 7× slower generation (33,8 → 4,7 tok/s)"Ollama already has a memory estimator. It started overestimating by 1.2 GiB, the model no longer fit, the scheduler sent 10% to the CPU, and generation dropped 7×. There is no need to argue the problem exists: the ecosystem already tried to solve it, and there is a quantified record of the failure.
ollama/ollama#14312 · score 19 · Q4_K_M"qwen2.5vl:7b uses significantly more RAM than qwen3-vl:8b despite smaller architecture and identical quantization"The reporter derives the theoretical KV cache from the architecture (28 layers, 4 KV heads, head dim 128 → ~0.44 GB @ 8k), measures the actual value (~2 GB when doubling the context), and arrives at a multiplier of 4–5× over theory. This is a human running Stargazer v2 by hand, and getting it wrong.
16 · Evidence
Three that delimit the scope
ggml-org/llama.cpp#27937 · the cost of a default"llama-tts: qwen3-tts default context (32768) allocates 3.5 GB of KV cache, 4096 would suffice"The reporter calculates the real need from the codec rate (12.5 tokens per second of audio): ~2,100 cells, versus 32,768 allocated, 3,584 MiB of KV cache, 7.1 GB in total. The cost comes neither from the model nor the hardware: it comes from a configuration default. Artifact in its purest form, trivially predictable if the predictor knows n_ctx.
ollama/ollama#16051 · M3 Ultra 96 GB · a declarable limit"MLX bf16 cold prefill is 60–400× slower than warm, even when peak memory fits in physical RAM"The reporter rules out the paging explanation: the 47.13 GiB peak fits comfortably in 96 GiB. The same (model, hardware, configuration) produces two costs separated by two orders of magnitude, depending only on the warm-up state. No static predictor solves this, and the paper is stronger for citing it by name.
ollama/ollama#12353 · 6 reactions · the product's request"Feature Request: Auto-size num_ctx to a user VRAM budget (and recalc on model switch)""Right now, users must guess a safe num_ctx. If it's too high, VRAM overflows and Ollama may slow down or fall back to CPU, with no clear feedback." This is the specification of Stargazer v2 written by a user, including recalculating on model switch and respecting the GGUF limit. Filter v1 discarded it as a "feature request".
17 · Threats to validity
Six declared biases
Concentration in Apple Silicon
Of the 307 issues with a specific platform, 201 (65%) are Apple Silicon. The conclusions are biased toward high-capacity unified memory, the opposite of the restricted Android/edge cohort. Android appears in 76, Jetson in 16, Raspberry Pi in 7.
Temporal asymmetry
2026 is 60% of the retained issues but 34.6% of the raw corpus. Measured cause: reporting practice matured, median prose doubled (914 → 1,896 chars), and metric mentions rose from 18.2% to 45.2%. Frequencies cannot be compared across years.
Repository concentration
llama.cpp + ollama = 73% of retained issues.
ROCm absent
HTTP 422 on all 300 queries. The study has 11 repositories, not 12; AMD coverage comes indirectly.
GitHub only
Forums, Discord, r/LocalLLaMA, and app stores are left out, likely where the energy cost, absent here, is perceived.
Mandatory metric
Requiring a number in the prose discards legitimate qualitative reports. Example: llama.cpp#3422 ("AMD GPU slower than CPU"), score 17, no number outside the log.
18 · Protocol
Labelling is what turns a heuristic into evidence
The filter is a set of rules we wrote while looking at the data. If the paper says "we found 620 cost issues" and the justification is "the script found them", the committee's response is immediate: why these weights? did you tune them until you got the number you wanted?
1Two independent evaluators: Tiago and Isaque/Breno, without mutual consultation (human round, to be run)
2Binary decision + written justification: mandatory for "no" decisions too
3Milestones at 5, 10, 20, 50, and 100%: early detection of protocol divergence
5Blind adjudication by the supervisor: the reported Kappa is the one from before adjudication
Kappa does not measure whether you got it right, there is no ground truth. It measures whether two people, reading the same definition, reach the same conclusion. That is what makes the study replicable.
Preliminary round over the 60 issues, conducted by LLM and not by humans: observed agreement of 90%, Kappa 0.795 (Substantial), 6 disagreements awaiting adjudication. Confirmed before adjudication: 22 of 60, which gives between 37% and 47% filter precision, depending on the adjudication.
83% of the disagreements come from a single ambiguity: does the definition require the hardware to be identifiable enough to reproduce the scenario? This question needs to be ratified before the definitive round.
19 · Repository literature
What the method rests on
Islam, Nguyen, Pan, Rajan (2019). A Comprehensive Study on Deep Learning Bug Characteristics. ESEC/FSE, 510–520.Base protocol. Vocabulary derived from symptoms, stratification by framework, funnel table. 2,716 posts + 500 commits, calibrates the order of magnitude of our N.
Islam, Pan, Nguyen, Rajan (2020). Repairing Deep Neural Networks: Fix Patterns and Challenges. ICSE.Relevance oracle. The existence of a fix is evidence that the defect is genuine, the origin of Gate 1 of our filter.
Codabux, Fard, Verdecchia, Palomba, Di Nucci, Recupito (2025). Teaching Mining Software Repositories. arXiv:2501.01903.Adherence to jargon. Recall in MSR is dominated by the mined community's vocabulary, the justification for tok/s, GGUF, Q4_K_M in the matrix. Also the record of rejected terms.
Thomas et al. (2024). muPRL: A Mutation Testing Pipeline for Deep RL based on Real Faults. arXiv:2408.15150.Two-axis strategy. Mining real repositories to build a failure taxonomy; the origin of the resource × device crossing.
Landis & Koch (1977) · Cohen (1960) · Byrt, Bishop & Carlin (1993).Agreement. Interpretation scale, the coefficient itself, and PABAK, which the report presents alongside Kappa to distinguish a bad protocol from a prevalence paradox.
Dantas Cruz, Paul, Song, Tilevich (2020). STARGAZER. IEEE SMDS.The origin work. Defines the regression targets (duration, energy, memory, CPU, throughput) that the keyword matrix needs to cover term by term.
20 · Repository literature
What supports the domain: and where the corpus disagrees
Imai, Nakazawa, Amaral, Choochotkaew, Chiba (2024). Predicting LLM Inference Latency: A Roofline-Driven ML Method. ML for Systems @ NeurIPS.Agrees. The KV cache as a component that grows with context, confirmed in 40% of retained issues, of which 90% are memory.
Cai et al. (2026). Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference. arXiv:2607.05475.Strongly agrees. NPU-specific bottlenecks, our largest lift in the entire corpus is QNN/Hexagon × NPU at 31.0.
Tu, Mishra, Dasari, Bourgeois, Wang (2026). Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices. arXiv:2607.21602.Agrees. Screening without execution on heterogeneous edge, it's the same problem; our scope cut (69% Artifact) delimits where this is viable.
Tummalapalli, Arayakandy, Pal, Kundan (2026). LLM Inference at the Edge: … Under Sustained Load. arXiv:2603.23640.Disagrees. The literature devotes papers to the thermal effect under sustained load; the corpus has 6 issues of Temperature, and the main one refutes the thermal cause.
Zou et al. (2026). EnerInfer: Energy-Aware On-Device LLM Inference. arXiv:2606.23001.Disagrees. Energy is a central axis in the literature and zero in the corpus as a primary category, despite battery, energy, power, and mAh being in the matrix.
Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115. · Wei et al. (2018). Singularity. ESEC/FSE.Models from the experiments' controlled variable; and v1's static-complexity tool, whose lack of an analogue for a weights file motivates v2.
The disagreement is a result. Energy and thermal cost is measured by researchers with instrumentation, but it is not reportable as a bug by practitioners. Either RQ1 narrows to memory/latency/throughput, or the study needs sources where the end user speaks.
21 · Synthesis
Six conclusions for Stargazer v2
Predict memory first
64% of issues have memory as the primary category, and latency/throughput appear 316 times as a secondary symptom of the same issues.
n_ctx is a mandatory feature
40% cite KV cache or context; 90% of those are memory. Estimating from the weights file is insufficient by construction.
Backend is a first-class variable
Backend × platform lift up to 31.0 against ~1.0 for model × hardware. The backend explains the variance that the model × hardware pair does not.
The accuracy bar is generous
With a median surprise of 4×, a predictor with 20–30% error already changes decisions. The target is the "fits / doesn't fit" boundary.
Declare the scope at ~26%
Labelling flipped the heuristic: Artifact is 26% of confirmed issues, Toolchain is 74%. The reach is smaller than it seemed; cite ollama#16051 by name.
Reformulate RQ2
Model × hardware does not concentrate problems in this corpus. Backend × platform does, and the backend is what the developer chooses.
22 · Next steps
Deliverables and what's missing
Done
• Traceable keyword matrix, with rejected terms
• 8-script pipeline, incremental and reproducible collection
• Corpus of 7,889 issues + 620 filtered + full ranking
• Funnel and table in the Islam et al. style, generated by script
• Analysis of the 620 issues and individual reading of the top 12
• Preliminary labelling round: Kappa 0.795, measured precision of 45%
data/msr/ · scripts/msr/ · docs/
Pending
○ Ratify the operational definition of RELATED, before the first line
○ Definitive labelling round of the 60 issues, current labels are preliminary
○ Label the remaining 560, or assume the 45% extrapolation
○ Decide on ROCm/ROCm: replace it or document the loss
○ Recalculate all frequencies over the labelled set
○ Draft the Study Design section from the generated artifacts
The structural findings, backend lift, median magnitude of 4×, absence of the energy axis, are robust because they do not depend on classification by category. The frequencies by category, however, do.
23 · To decide together
Questions for Breno and Isaque
Does the definition of RELATED require the hardware to be identified precisely enough to reproduce the scenario? : 83% of the pilot's disagreements come from this single ambiguity.
Does RQ1 narrow to memory/latency/throughput, or do we seek end-user sources to recover the energy axis?
Does RQ2 get reformulated around backend × platform, since model × hardware doesn't hold up statistically?
Do we replace ROCm/ROCm with another repository from the AMD ecosystem, or document the loss of coverage?
Does Kappa go into the paper as inter-rater reliability? If so, labelling needs to be done without looking at the machine pre-annotation.