§1Executive summary and maturity status
The study mines GitHub Issues from 11 repositories in the on-device inference ecosystem to measure what practitioners report when a model's execution cost surprises them. It exists to answer, with observed data rather than assumption, whether the cost predictor Stargazer v2 proposes solves a real problem, and how big that problem is.
How to read this document: what is definitive and what is not
Every claim in the report carries one of these three seals. They are not decoration: they separate what can already go into the paper from what still depends on pending human work.
| Seal | Means | Where it appears |
|---|---|---|
| measured | Derived directly from the collected data, by a reproducible script, with no intervening human judgment. | Funnel counts, collection provenance, discard statistics, backend lifts, magnitude of degradations. |
| preliminary | Depends on labels that exist, but were produced in a pilot round and will be replaced. | Kappa of 0.795, precision of 45%, entry-point and category distribution over the confirmed set; M1–M3 pilot screening (86 included, Kappa of 0.731). |
| pending | The instrument is built and the sample drawn, but no one has filled it in yet. | Filter recall (audit of 45 discards), human round of M1–M3 screening of the 172 approved repositories. |
The labelling round that produced the Kappa of 0.795 was not run by two independent humans. It was run by two annotator roles operated by an LLM, defined in .claude/skills/msr-avaliador-1 and msr-avaliador-2, each with a deliberately distinct decision rule (detail in §3.6).
Exact consequence: the Kappa measures agreement between two written decision rules, not between two human judgments. This has real value — it is a dry run that already exposed a serious pipeline defect (§5.2) and an ambiguity in the operational definition itself (§6.4) before spending hours of human labelling. But this is not the Kappa that goes into the paper. The definitive round (Tiago + Isaque/Breno, in parallel and without consultation) is pending, and every frequency derived from it must be recalculated.
The six conclusions the corpus supports today
- Memory is the cause; speed is the symptom. 64% of retained issues have Memory as the primary category, and latency/throughput appear 316 times as the secondary category of those same issues. A predictor that gets memory right solves most of the problem.
- The median surprise is a factor of 4. This is not a calibration error, it is a qualitative prediction failure. The accuracy bar required of Stargazer v2 is generous: what is at stake is "fits / doesn't fit".
- RQ2, as written, has a negative answer. Model × hardware does not concentrate problems (lifts between 0.54 and 1.54). Backend × platform does concentrate them (lifts up to 31.0). The RQ needs to be reformulated.
- The energy axis does not survive the filter, and that is different from not existing. Zero Energy issues in the retained set, but 77 discarded issues mention battery, energy, power, or thermal. The cause is the measurability criterion, not the source.
- The predictor's declarable reach is ~26%, not 69%. Labelling reversed the heuristic's estimate: the dominant slice is Toolchain (regression between versions, backend choice, scheduler decision), which no static predictor anticipates.
n_ctxis a mandatory input feature. 40% of retained issues mention KV cache or context, and 90% of those are memory issues. Estimating from the weights file alone is insufficient by construction.
§2Framing: why MSR, and what it answers
The role of MSR within the paper
The Stargazer v2 bench experiments (llama.cpp on Linux laptop and Android, Qwen2.5 at three sizes, fixed vs. marginal cost separation, Roofline validation) measure cost. They do not say which costs matter in practice, nor how often practitioners misjudge them in advance. MSR occupies exactly that gap: it is the observational evidence that the problem exists outside the lab, and it is the source of the evaluation instances against which the predictor will be tested.
The claim the corpus needs to support is narrower and stronger than "people suffer from inference cost":
"Running an LLM on-device consumes a lot of memory and is slow." True, banal, and does not support a software-engineering paper.
"Execution cost is systematically unpredictable before execution, the prediction error has a median magnitude of 4×, and the ecosystem has already tried to solve it and failed in a documented way." This is searchable, quantifiable, and specific.
The three research questions and how each was instrumented
| RQ | Question | Instrument in the pipeline | Status |
|---|---|---|---|
| RQ1 | What categories of resource problem are reported when running LLMs on-device? | Column categoria_recurso ∈ {Memory, Latency, Energy, Temperature, Throughput, Other}, assigned by symptom regex and corrected during labelling. |
answered with a relevant negative result on the energy axis |
| RQ2 | Which model × hardware × backend combinations generate the most problems? | Triple extracted by regex over model name, SoC/platform, and backend; concentration test by lift over the marginal frequencies. | negative as written, requires reformulation to backend × platform |
| RQ3 | What is the impact of quantization parameters on reported execution cost? | Fields quantizacao and metricas_numericas; terms GGUF, KV cache, context length in the search matrix. |
answered with a counter-intuitive finding (quantize the cache, not the weights) |
Source: data/msr/keyword_matrix.md, section 6.1 (keyword → RQ map).
§3Methodology, stage by stage
The pipeline has seven stages. Each has a hand-written input artifact (the source of truth), a script that runs it, and a generated output that is never edited by hand. The rule is deliberate: if the number in the paper diverges from the number in the dataset, the divergence is a defect, not a wording detail.
| # | Stage | Script | Source of truth (hand-written) | Generated output |
|---|---|---|---|---|
| 0 | Repository selection | select_repos.py | repository_matrix.md | repository_selection.md, repo_candidates.csv |
| 1 | Mining | mine_issues.py | keyword_matrix.md | issues_raw.jsonl, mining_provenance.json |
| 2 | Quality filter | filter_issues.py | weights in VIOLATION_SIGNALS | issues_filtered.jsonl, filter_stats.json, top_issues.md |
| 3 | Sampling | sample_for_annotation.py | seed 2028, quota per repository | issues_sample_eval.csv |
| 4 | Labelling | split_for_annotators.py, merge_annotations.py | annotation/PROTOCOL.md | annotator_1.csv, annotator_2.csv |
| 5 | Agreement | calculate_kappa.py | Kappa threshold ≥ 0.61 | kappa_report.md, disagreements.csv |
| 6 | Discard audit | inverse_filter.py | declared suspicion weights | discard_audit.md, audit_sample.csv |
| 7 | Funnel | build_funnel.py | n/a | funnel_diagram.md, funnel_table.md |
3.1 Repository selection: the funnel that comes before the funnel
The first version of the study mined 11 repositories chosen from knowledge of the ecosystem. The justification ("each one is a distinct entry point") is true, but it was built after the choice, which makes it a purposive sample with no declared population frame. Without a protocol, the answer to "how did you choose these repositories?" is "the ones we knew".
The protocol was then written and executed against the API. It declares the unit of analysis (a public GitHub repository), the frame (repositories that implement, package, or consume LLM inference on an end-user device), and what falls outside it by definition: training, datacenter serving, libraries with no on-device path, link collections.
- A1 · A2
- not a fork; not archived
- A3
has_issues == true- A4
- ≥ 50 closed issues, funnel viability threshold
- A5
- ≥ 20 issues updated in the last 12 months
- A6
- ≥ 200 stars, relevance proxy
- M1
- does the repository run or package LLM/ASR inference?
- M2
- is there a documented path to an end-user device?
- M3
- does the tracker receive execution-cost reports, not just build reports?
Only repositories with M1 and M2 true enter the study. M3 false is a declared caveat, not an exclusion — density of cost reports is a result, not a criterion.
A5 measures tracker activity, not code activity. The initial operationalization used pushed_at, which counts commits. A repository that exists only to receive issues — the bug tracker of a closed-source product — has no commits. Since the study mines issues, the correct measure of activity is the tracker's. The change was motivated by an observed failure: lmstudio-ai/lmstudio-bug-tracker failed on pushed_at despite having 489 closed issues and an active tracker. The criterion was poorly operationalized, not poorly calibrated.
A6 is the most questionable of the six criteria. Stars favor Western, English-speaking projects. This is declared, and the protocol opens an explicit exception for repositories that fill an empty coverage cell.
Selection funnel as executed
| Stage | Criterion | Repositories |
|---|---|---|
| E0 | raw results from the 12 seed queries | 3,430 |
| E1 | unique repositories | 3,175 |
| E2 | A1 + A2 · not a fork, not archived | 3,075 |
| E3 | A3 · issues enabled | 3,029 |
| E4 | A6 · stars ≥ 200 (+0 by cell exception) | 508 |
| E5 | A5 · tracker active in the last 12 months | 246 |
| E6 | A4 · ≥ 50 closed issues | 172 |
| E7 | manual screening M1–M3 human round pending | 0 |
Source: data/msr/repository_selection.md, generated on 2026-10-05. A6 runs before A5 and A4 because the latter cost API calls and, applied last, run over fewer candidates. The 172 approved are in annotation/repo_screening.csv; the blind screening protocol is in annotation/SCREENING_PROTOCOL.md.
To test stage 00 end to end before the human round, two LLM-operated evaluator roles filled in the two blind spreadsheets. Both included 86 of the 172, excluded 63, and disagreed on 23, still without a tiebreak. The pre-tiebreak Kappa was 0.717 on M1, 0.827 on M2, and 0.731 on the inclusion decision. The 86 are the list that has been under collection since 2026-10-07 (see the funnel page), but the pilot's provenance file prohibits citing these numbers in the paper. In the official funnel, E7 remains 0 until human screening.
A1–A6 measure popularity and activity and never ask whether the repository is about on-device inference cost. The context signal asks that question using the vocabulary of the keyword matrix, the same one stage 01 uses for mining. The two halves are measured in different units and are therefore reported separately:
- 115 of 172 have at least one on-device cost issue in the tracker, by the matrix's 3 primary queries;
- 5 of 172 mention, in their own name, description, or topics, a resource term and a device term.
Using it as a threshold would repeat A6's mistake: a single number eliminating the scarcest cohort. It only sets the order of the screening spreadsheet (tracker probe, then text, then issue volume).
| Stars ≥ | Pass A6 | Approved | Of the 12 |
|---|---|---|---|
| 100 | 661 | 184 | 12 |
| 200 | 508 | 172 | 11 |
| 500 | 315 | 151 | 11 |
"Approved" pass A1–A6; "Of the 12" counts the original repositories that remain approved. Multiplying the threshold by five (100 → 500 stars) changes the approved set by 33 repositories (184 → 151). Of the original 12, only lmstudio-bug-tracker depends on where the line falls.
A distinction that matters for the paper, and that the validation document states explicitly: the protocol was applied to the study's 12 repositories in a confirmatory way — it checked the list against the criteria after the fact. It did not generate the list. In the generative funnel, 11 of the 12 survive:
lmstudio-ai/lmstudio-bug-trackerfails A6 (155 stars). In the retroactive validation it enters through the cell exception, being the onlypapel:aplicativo(app-role) entry in the original list; in the generative funnel that cell already has 29 candidates, so the exception does not apply.ROCm/ROCmsurvives under the name GitHub resolves it to today,ROCm/legacy-rocm-build(HTTP 301 redirect). Comparing only the declared names would mark it as failed. Under the new name, the pilot collection of the 86 obtained 304 issues, which points to the old name as the cause of the collection's HTTP 422 (§3.3).
The claim that is actually supported is therefore "the study's repositories satisfy declared quality criteria", not "the protocol selected these repositories". The generative funnel produces a different list: there are 161 approved candidates that the original list did not have, and the pilot screening included 75 of them.
The original list leaves cells empty: the retroactive validation (data/msr/repository_validation.md) flags 5 of 18 — plataforma:web · acelerador:qualcomm · acelerador:intel · acelerador:nvidia · acelerador:arm_cpu. Among the 172 approved, all 18 have at least one repository — for example 29 in papel:aplicativo, none of them dependent on the star exception, and 2 in acelerador:qualcomm.
These assignments are automatic suggestions, not triaged classification: the equivalent classifier at the issue level was wrong by almost a factor of three. The bias report (data/msr/selection_bias_report.md) uses these suggestions to test concentration in Apple: it drops on the accelerator axis (50% → 46%) and does not drop on the platform axis (33% → 44%). This is a finding to report, not a reason to adjust the queries until the number flips. Until human screening, every empty cell in the original list remains a threat to external validity to be declared.
3.2 Search strategy: no term enters without a reference backing it
The keyword matrix is the single source of truth for the search, and mine_issues.py reads the file: changing the queries there changes the collection. The property this matrix guarantees is the answer to the committee question "why exactly these keywords?" — each of the 26 has a line with a technical justification and a literature reference.
The vocabulary was derived through four rules, adapted from Islam et al. (2019)'s protocol for Deep Learning bugs to on-device inference cost:
- Seed from the symptom literature. Islam et al. derive the vocabulary from observable symptoms, not root-cause names, because the reporter describes what they saw, not the diagnosis. Hence
"out of memory",OOM,slow,thermal throttl, and not "allocator memory pressure". - Seed from the study object's metrics. Every Stargazer regression target (duration, energy, memory, throughput) needs a corresponding term, otherwise the evidence does not cover the variable the model predicts.
- Seed from the ecosystem's native jargon. Codabux et al. (2025) warn that MSR recall is dominated by adherence to the mined community's jargon, not the academic one. Hence
tok/s(not just "throughput"),KV cache,GGUF,Q4_K_M. - Precision filter by device context. Pure resource terms return massively server-side issues. Every Resource term is combined with at least one Device/Context term, the two-axis strategy.
A term is rejected if more than 50% of the first 20 results are build/compilation errors or conceptual questions. Reporting the rejections is part of the replicability protocol. The five on record:
| Rejected term | Measured reason |
|---|---|
crash | 17/20 were build failures or segfaults with no resource metric at all |
performance | 14/20 were feature requests ("performance roadmap"), not measured-cost reports |
CUDA | anchors the result in datacenter/desktop dGPU, the cohort the RQ excludes |
bug | orthogonal to the cost axis; does not discriminate |
temperature | collides with the LLM sampling hyperparameter temperature; replaced with thermal + throttl |
The API constraint that determined the search design
Verified against the official REST documentation on 2026-09-29:
| Constraint | Value | Design implication |
|---|---|---|
| Query length | ≤ 256 chars | Not the bottleneck — ours run 71–90 characters |
| Boolean operators | ≤ 5 | The real bottleneck. Limits each query to ~6 terms |
| Results per query | 1,000 | Requires decomposition by repo: to avoid truncation |
| Authenticated rate limit | 30 req/min | Backoff mandatory, 105 waits during actual collection |
Recorded methodological consequence: the 26 keywords do not fit into 3 queries under the 5-operator cap.
The solution has two layers, both executed and deduplicated by (repo, issue_number):
- Three primary queries, high-precision and reportable in the paper, each crossing a resource axis with a device cohort: memory × mobile; throughput/latency × Apple and GPU-equipped edge; thermal/energy × NPU, mobile, and embedded.
- One
KEYWORD-GRID: Cartesian product of 20 resource terms × 15 device terms, one atomic query per pair, 0 to 2 operators each. This is what gives full matrix coverage, and it is what explains the 3,600 queries executed (300 per repository × 12).
3.3 Mining: full provenance and two handled anomalies
/search/issuesPer-repository provenance, with new and revisited issues per query, in data/msr/mining_provenance.json.
ROCm/ROCm returned nothing
The repository's 300 queries all failed with HTTP 422 ("the resources do not exist or you do not have permission"). The study therefore has 11 repositories, not 12. AMD coverage comes indirectly from llama.cpp and ollama, and indeed ROCm/HIP appears in 96 of the retained issues. Cause identified later (§3.1): the repository was renamed to ROCm/legacy-rocm-build, and under that name the pilot collection of the 86 obtained 304 issues. TARGET_REPOS still needs to be updated to the resolved name.
All five were ggml-org/llama.cpp combined with GGUF. Since the ordering is created desc, truncation always cut off the old tail. They were redone with recursive date-range bisection (mine_issues.py --requery-truncated, 98 requests, 6 minutes): +278 issues, 89% of them from 2023–2024, exactly the distribution the hypothesis predicts, which validates the fix.
Corpus: 7,611 → 7,889. This is the number used throughout the report. Where 7,611 appears in old annotations, it is the pre-bisection count.
3.4 Quality filter: the v1 criterion was replaced, and the measurement of why
The two criteria run over the same raw collection; only the filter changed. v1 is frozen in data/msr/baseline_filter_v1/ and remains reproducible with filter_issues.py --criterion v1.
v1 required model AND hardware AND numeric metric, searched for anywhere in the text. Measuring against the real corpus revealed the defect: llama.cpp's Eval bug template has the reporter paste the entire startup output, and 38% of that repository's raw issues carry this signature. The pasted log, on its own, satisfies all three gates:
| v1 gate | What the startup log provides for free, without the issue being about cost |
|---|---|
| Numeric metric | 0.15 MiB, 1941.02 MiB, 1187.01 ms from the load table |
| Model | the model name appeared in 88% of retained issues |
| Hardware | build flags (NEON, aarch64, Metal), 57.6% of retained issues had only generic hardware |
Aggravating factor: the pattern \bv?ram\b matched the VRAM that the runtime prints in every load output, dumping template, tokenizer, and kernel issues into the Memory category.
v2 does not look for "the issue mentions cost". It looks for "the issue records a cost expectation that was violated". The difference is operational, not rhetorical: an issue that passes this criterion carries both the expected and the observed value — in other words, it is already a test case for the predictor.
The three concrete changes
- Separate prose from log (
extract_prose). Every regex now runs only over what the reporter wrote: fenced code blocks,<details>, HTML comments, runtime log lines, andkey = valuedumps are stripped out. This is the highest-impact fix. - Change what is searched for. The expectation may be in the template's "What did you expect to happen?" section, or in markers such as
theoretical … observed,2x slower than,since build bNNNN,NPU slower than CPU. - The scope gate became a penalty, not an exclusion. Measured: of v1's 1,002 cloud-cohort exclusions, 779 were merely consumer GPUs (desktop RTX), not datacenter. A personal desktop is a legitimate cohort, and the content of those issues (KV cache, quantization, scaling with context) answers RQ3 directly. In v2, only unambiguous datacenter is a hard exclusion; consumer GPU, build error, and feature request become score penalties (−2, −5, −3).
| Dimension | Criterion v1 | Criterion v2 |
|---|---|---|
| Criterion | model AND hardware AND metric, in the whole text | violated cost expectation, in the prose |
| Retained issues | 1,642 (21.6%) | 620 (7.9%) |
| Resource symptom in the title | 16.7% | 49.1% |
| Estimated precision (sample of 40, judged by hand) | ~17% | ~60% |
| Precision measured by labelling | n/a | 45% |
| Hard exclusions | 3 (build, question, cloud) | 1 (datacenter cohort) |
The two versions are not nested: v2 keeps 332 of v1's retained issues, discards 1,310, and recovers 277 that v1 had rejected. Source: data/msr/filter_v1_v2_comparison.md.
v2's three filters, in the order they are applied
Filter 1, measurability (hard condition). Requires at least one number with a unit (MB, GB, tok/s, ms, °C, mAh, W, %) in the prose. Without a number measured by the reporter, the issue cannot serve as an evaluation instance for a cost model. Discards 4,336 issues as the primary reason (4,412 in total lack a metric in the prose).
Filter 2, violated-expectation score. Sum of weighted signals; retains from score ≥ 10 up. The weights are all declared in VIOLATION_SIGNALS:
| Signal | Weight | What it matches |
|---|---|---|
teoria_vs_observado | +3 | theoretical, computed, derived from, multiplier, observed … vs/but/however |
expectativa_explicita | +2 | expected, supposed to, should be/take/use/need/fit |
regressao_versao | +2 | since commit/version/bNNNN, worked fine in/before, regress*, bisect* |
razao_inesperada | +2 | N× slower/faster/more, twice as, half the |
backend_pior_que_baseline | +2 | npu|gpu|metal|vulkan|qnn|… slower than … cpu, falls back to cpu |
escala_com_contexto | +2 | kv cache, n_ctx, context length/size/window |
comparacao_entre_configs | +1 | more/less/slower than, vs, compared to, despite, identical |
Cutoff sensitivity
| Cutoff | Issues with score ≥ cutoff |
|---|---|
| ≥ 6 | 2,820 |
| ≥ 8 | 1,584 |
| ≥ 10 (adopted) | 758 |
| ≥ 12 | 298 |
| ≥ 14 | 119 |
| ≥ 16 | 38 |
Counts before scope exclusions.
Filter 3: scope (the only hard exclusion)
Unambiguous datacenter cohort, A100, DGX, EC2, Kubernetes, tensor parallel: 224 issues excluded. This is a scope boundary (the study is about on-device execution), not a quality signal.
Distribution: llama.cpp 133 · ollama 81 · onnxruntime 7 · others 3.
Discard reasons, in total
| Reason | Issues | % of discards |
|---|---|---|
SEM_MENSURABILIDADE | 4,336 | 59.7% |
SCORE_BAIXO | 2,709 | 37.3% |
EXCLUSAO | 224 | 3.1% |
The score is triage, not truth. It orders the corpus for manual labelling, which remains the study's actual measure. It also discards legitimate qualitative reports: llama.cpp#3422 ("AMD GPU slower than CPU") scores 17 and falls out for lacking a number outside the log. This is a conscious loss — without a number, the issue cannot serve as an evaluation instance for a cost model.
3.5 Sampling: stratified, proportional, with a fixed seed
Of the 620 filtered issues, 60 were drawn by stratified proportional sampling by repository, with a per-stratum floor, replicating the framework-stratification procedure of Islam et al. (2019). The seed is fixed (--seed 2028), which makes the sample exactly reproducible.
The row order is shuffled with the same seed, and this serves a control function: the Kappa milestones at 5%, 10%, 20%, 50%, and 100% are prefixes of the spreadsheet. If the 10% milestone's Kappa is already low, the protocol alignment meeting happens before spending the entire labelling effort. The protocol explicitly instructs: "stop at row 20 and run Kappa" — 20 rows lost instead of 60.
- Minimum Kappa
- ≥ 0.61, the Substantial band of Landis & Koch (1977)
- Minimum n
- 20 for a milestone to be considered reliable — below that, a single disagreement shifts the Kappa by more than 0.2
- If it fails
- mandatory alignment meeting before continuing labelling
3.6 Labelling: the operational definition and the two annotators
The definition of RELACIONADO is the most important piece of the study, because it is what converts a text corpus into evidence. It requires all four conditions simultaneously:
- There is a cost expectation that was violated. Explicit ("expected ~30 tok/s"), derived ("the model is smaller, it should use less RAM"), or comparative ("2× slower since build bNNNN").
- The deviation is quantified. At least one number with a unit, written by the reporter about their own run. A number copied from a third-party benchmark does not count.
- The run is on-device. Mobile, embedded, laptop, or consumer desktop. Datacenter does not count.
- The deviation is attributable to a model, a configuration (quantization, context, backend), or an identifiable platform.
Missing any one → 0. This includes issues that are about cost but without a violated expectation (an optimization request), or functional bugs that merely mention numbers in a pasted log.
Edge cases decided before the round
| Situation | Decision | Reason |
|---|---|---|
| Performance regression between versions, with a number | 1 | the expectation is the previous version |
| Accelerator (NPU/GPU) slower than the CPU, with a number | 1 | the expectation is that the accelerator is faster |
| Feature request that exists because cost is unpredictable | 1 | direct evidence of the problem |
| "It's slow" with no number | 0 | cannot serve as an evaluation instance |
| Crash with a log that contains MiB | 0 | incidental number, not a measurement |
| OOM on an A100 in a cluster | 0 | outside on-device scope |
The two annotators have deliberately different decision rules. This is not an oversight: it is the design. If both applied the same criterion, the Kappa would only measure reading noise. Applying distinct criteria, it measures whether the definition is robust to two defensible readings of what counts as evidence.
Wrote the protocol. Responsibility: fidelity to the written definition, not instinct about what "seems" relevant. Question: is there a cost expectation, stated or derivable, and a quantified deviation from it?
Labelled 27 as 1, 33 as 0.
Did not write the protocol. Assesses whether the dataset supports evaluating the predictive model. Question: can a triple be extracted (model, hardware/platform, configuration) and an observed cost that the predictor would have to estimate without running?
Labelled 23 as 1, 37 as 0, systematically stricter (see §6.4).
- Every disagreement goes into
disagreements.csv. - The supervisor decides without seeing who labelled what.
- The decision goes into
final_decision;discrepancy_resolvedbecomesTRUE. - If the same type of disagreement appears 3 times or more, the operational definition is amended, and the amendment is recorded with its date. triggered, see §6.4
- The Kappa reported in the paper is the one at the 100% milestone before the tiebreak. After the tiebreak it is 1.0 by construction and measures nothing.
3.7 Discard audit: the half of the problem the funnel doesn't measure
The pipeline measures precision: of the retained issues, how many are genuine. On its own, it does not answer the complementary question, of the genuine issues that exist in the corpus, how many did the filter lose? Without that there is no answer to "what did you throw away?".
inverse_filter.py scores each discard by false-negative suspicion signals, with declared weights: sintoma_no_titulo +3, magnitude_explicita +3, eixo_ausente_no_retido +2, score_marginal (8–9) +2, hardware_especifico +1, engajamento_alto +1, metrica_apenas_no_log +1, against bug_correcao_no_titulo −3, build_no_titulo −3, cohort_datacenter −4.
eixo_ausente_no_retido is overweighted on purpose. The retained set has no Energy issues at all and only 6 Temperature issues; if the filter is blind to that axis, the discard pile is the only place it would show up. The overweight is a declared decision, not a hidden bias.
| Suspicion stratum | Range | Issues | In the audit sample |
|---|---|---|---|
| high | ≥ 6 | 264 | 22 |
| medium | 3–5 | 1,019 | 15 |
| low | ≤ 2 | 5,986 | 8 |
| total | n/a | 7,269 | 45 |
The allocation is not proportional: half the sample comes from the high-suspicion stratum, because proportional sampling would put almost everything in the low stratum, where the false-negative rate is near zero. The decisao column of the 45 rows is empty — the audit is a built instrument that has not yet been run.
The post-stratification recall estimation formula is already declared, so the result is not chosen after seeing the data:
FN = Σstratum ( hits / sampled × stratum_size )
TP = 620 retained × 45% precision ≈ 279
recall = TP / (TP + FN)
Example reading: if the audit finds false negatives in 20% of the high stratum, 5% of the medium, and 0% of the low, then FN ≈ 104 and recall comes out around 73%.
§4The funnel, end to end
Four stages, from 7,889 to 27. The bars show retention relative to the previous stage, not the absolute proportion, which would make the last stages disappear.
KEYWORD-GRID · 3,600 queries over 11 repositories · issues created between 2017-12-03 and 2026-09-30
Extrapolating the 45% precision to the 620 filtered issues: about 279 genuine issues in the retained set.
Funnel by repository
Format adapted from Table 1 of Islam et al. (2019). The retention column reveals a difference in reporting culture between communities, not in software quality.
| Repository | Raw | Filter | Retention | Sample | Confirmed | % Related |
|---|---|---|---|---|---|---|
ggml-org/llama.cpp | 3,863 | 330 | 8.5% | 23 | 10 | 43% |
ollama/ollama | 1,747 | 124 | 7.1% | 10 | 5 | 50% |
mlc-ai/mlc-llm | 434 | 44 | 10.1% | 5 | 1 | 20% |
pytorch/executorch | 433 | 18 | 4.2% | 3 | 1 | 33% |
microsoft/onnxruntime | 376 | 28 | 7.4% | 4 | 2 | 50% |
google-ai-edge/mediapipe | 322 | 5 | 1.6% | 2 | 1 | 50% |
lmstudio-ai/lmstudio-bug-tracker | 297 | 35 | 11.8% | 4 | 2 | 50% |
ggml-org/whisper.cpp | 134 | 5 | 3.7% | 2 | 1 | 50% |
ml-explore/mlx | 108 | 20 | 18.5% | 3 | 3 | 100% |
google-ai-edge/LiteRT | 102 | 9 | 8.8% | 2 | 1 | 50% |
apple/coremltools | 73 | 2 | 2.7% | 2 | 0 | 0% |
| TOTAL | 7,889 | 620 | 7.9% | 60 | 27 | 45% |
mlx has the highest density of cost reports (18.5%) and mediapipe the lowest (1.6%). The reading is about reporting culture: mlx is used by people who measure; mediapipe gets a lot of integration issues.
Characterization of the retained corpus
§5Results and conclusions
5.1 Labelling reliability
Cohen's Kappa at the 100% milestone came out at 0.795, the Substantial band of Landis & Koch, above the 0.61 acceptance criterion declared before the round. The earlier milestones were all 1.000, which indicates the disagreement is concentrated in the second half of the spreadsheet (which, being shuffled, has no thematic ordering — so this is judgment disagreement, not warm-up noise).
Final milestone diagnostics
| Metric | Value | Reading |
|---|---|---|
| Cohen's Kappa | 0.795 | Substantial |
| 95% CI | [0.64; 0.95] | lower bound above the criterion |
| Observed agreement (p₀) | 90.0% | raw agreement rate |
| Expected agreement (pₑ) | 51.2% | agreement attributable to chance |
| Prevalence index | 0.167 | < 0.6 → no deflation from a rare class |
| Bias index | 0.067 | < 0.2 → no severe systematic bias |
| PABAK | 0.800 | Kappa adjusted for prevalence and bias |
| Standard error | 0.079 | asymptotic |
Prevalence and bias are reported because the Kappa paradox (high agreement with low Kappa) is exactly the risk posed by unbalanced samples. Here both indices fall in the safe range, which authorizes reading 0.795 as real agreement.
Confusion matrix
| Ann. 2 = NO | Ann. 2 = YES | |
|---|---|---|
| Ann. 1 = NO | 32 | 1 |
| Ann. 1 = YES | 5 | 22 |
The asymmetry is informative: 5 of the 6 disagreements are Annotator 1 accepting and Annotator 2 rejecting. This is not noise, it is a systematic difference in criteria, analyzed in §6.4.
Precision by score range
| Range | Confirmed / sampled | Precision |
|---|---|---|
| 10–11 | 13 / 33 | 39% |
| 12–13 | 7 / 19 | 37% |
| ≥ 14 | 7 / 8 | 88% |
The score discriminates, but almost all of its power is at the top — between 10 and 13 the precision is indistinguishable from constant. So: to assemble the set of evaluation instances, use score ≥ 14 (119 issues, ~88% precision); to estimate population frequencies, use the cutoff of 10 with the 45% correction.
5.2 The serious correction: automatic entry-point classification is invalid
This result alone justifies the existence of manual labelling, and it is the kind of finding that needs to appear in the report even (especially) when it contradicts a claim we made earlier.
| Entry point | Heuristic (620) | Labelled (27) | Reading |
|---|---|---|---|
| Artifact, the cost comes from a model property | 69.2% | 25.9% (n=7) | overestimated by a factor of ~2.7 |
| Toolchain, comes from the runtime, compiler, or backend | 23.7% | 74.1% (n=20) | underestimated; it is the dominant slice |
| Platform, comes from the device's physical envelope | 7.1% | 0.0% (n=0) | does not appear in the confirmed set |
The heuristic classified any issue mentioning quantization, GGUF, or KV cache as Artifact. But 66% of retained issues mention quantization, almost always as context for the report, not as the cause of the cost. It is the same disease as the v1 filter: matching on an incidental mention rather than on the issue's actual claim.
Concrete example: llama.cpp#29063 ("VRAM usage increased from b10680, SYCL backend") is a regression between builds, Toolchain. The heuristic marked it Artifact because the body cites Q4_K_M.
The claim "69% of the corpus is, in principle, a function of artifact properties", which was in the earlier version of the analysis and in the slides, is wrong. The measured value is 26%, and the dominant slice is Toolchain: regressions between versions, backend choice, scheduler decisions. None of that is predictable from the weights file.
This narrows Stargazer v2's declarable reach and needs to be stated that way. With n = 27, the 95% CI for the 26% is wide (≈ 11%–46%), but the inversion relative to 69% falls well outside it: the qualitative conclusion is solid; the point estimate is not.
In the opposite direction, the resource category classification holds up reasonably well, which bounds the defect: the problem is in the origin taxonomy, not the symptom one.
| Category | Heuristic (620) | Labelled (27) | Reading |
|---|---|---|---|
| Memory | 63.9% | 55.6% | dominant in both; moderate overestimation |
| Latency | 12.3% | 22.2% | was underestimated |
| Throughput | 10.8% | 22.2% | was underestimated |
| Temperature | 1.0% | 0.0% | n/a |
| Energy | 0.0% | 0.0% | the finding survives labelling |
| Other | 12.1% | 0.0% | all rejected, independent validation |
Outro (Other) disappearing entirely from the confirmed set is a useful cross-validation: the heuristic was already signaling that it did not know how to classify those issues, and labelling confirmed they were not evidence.
5.3 RQ1: memory is the cause, speed is the symptom
| Category | As primary | % | As secondary |
|---|---|---|---|
| Memory | 396 | 63.9% | n/a |
| Latency | 76 | 12.3% | 205 |
| Other | 75 | 12.1% | n/a |
| Throughput | 67 | 10.8% | 111 |
| Temperature | 6 | 1.0% | 9 |
| Energy | 0 | 0.0% | 6 |
Memory dominates as the primary category (64%), and latency/throughput appear 205 and 111 times as the secondary category of those same issues. The recurring pattern is causal, not coincidental: a memory-estimation error pushes part of the model onto the CPU or triggers reallocation, and the user perceives this as slowness.
Design consequence: a predictor that gets memory right solves most of the problem, because observed latency is frequently a function derived from where the model ended up fitting.
5.4 The magnitude of the cost surprise
81 issues (13% of retained issues) report an explicit magnitude of degradation. This is the corpus's strongest descriptive statistic, because it depends on no classification at all — it is the reporter saying "N× slower" or "N% worse".
| Ratio range | Mentions |
|---|---|
| 1.1–2× | 10 |
| 2–3× | 25 |
| 3–5× | 27 |
| 5–10× | 23 |
| 10–100× | 22 |
| >100× | 5 |
Percentage degradation, when reported in % rather than as a ratio: median of 33%.
When a practitioner is surprised by cost, the median surprise is a factor of 4. This is not a calibration error, it is a qualitative prediction failure. A cost model that was off by 20% would be irrelevant next to this; what is at stake is the difference between "fits" and "doesn't fit", between "usable" and "unusable".
Corollary for evaluating Stargazer v2: the accuracy bar is generous. The target is not fine numeric precision, it is getting which side of the boundary the configuration falls on. And this number comes from the data, not from assumption.
5.5 RQ2: negative result, and where the variance actually is
RQ2 asks which model × hardware combinations generate the most problems. The test is a lift test: the observed frequency of the pair divided by the expected frequency if model and platform were independent. Lift ≈ 1.0 means the pair is simply the product of the marginal frequencies — that is, there is no real concentration.
Common linear scale, maximum 31.0. The red dashed line marks lift = 1.0 (independence). Blue bars: the backend × platform axis. Gray bars (red label): the model × hardware axis, all hugging the independence line.
All model × hardware lifts fall between 0.54 and 1.54, indistinguishable from independence. The apparently "hot" pairs are an artifact of the marginals: llama appears in 421 of the 620 issues and apple-silicon in 201, so the llama × Apple Silicon pair with n=111 is exactly what would be expected by chance.
Reporting this is worth more than forcing a ranking of pairs the test does not support.
Lifts of 31.0 and 24.8 against ~1.0. The concentration of problems is backend-specific, not model-specific. The reformulated version, "which backend × platform combinations concentrate cost problems?", has an answer, is statistically supported, and is more actionable: the developer chooses the backend, not the SoC.
Distribution by backend (74% of issues name at least one)
| Backend | n | % | Dominant category within the backend |
|---|---|---|---|
| CUDA | 200 | 32.3% | Memory (67%) |
| Vulkan | 134 | 21.6% | Memory (70%) |
| Metal | 116 | 18.7% | Memory (57%) |
| ROCm/HIP | 96 | 15.5% | Memory (79%) |
| MLX | 67 | 10.8% | Memory (49%) |
| WebGPU/wasm | 35 | 5.6% | Memory (51%) |
| SYCL/oneAPI | 23 | 3.7% | Memory (87%) |
| QNN/Hexagon | 20 | 3.2% | Memory (50%) |
| XNNPACK | 15 | 2.4% | Latency (47%) |
Memory dominates within every backend (49–87%), independent confirmation of Finding 1. XNNPACK is the only exception, and it is precisely the mobile-CPU backend, where there is no cross-device memory transfer to get wrong. The exception reinforces the mechanism rather than weakening it.
5.6 RQ3: the problem isn't quantizing the model, it's quantizing the cache
| Signal | n | % of retained |
|---|---|---|
| Mention a quantization scheme | 409 | 66% |
| Mention KV cache / context | 249 | 40% |
| Of those mentioning KV cache, are Memory | 224 | 90% |
Most-cited schemes: q8_0 (127), q4_k_m (122), f16 (81), q4_0 (64), bf16 (39), fp16 (36).
q8_0 leads, and appears more often as a KV cache type than a weights type. The highest-magnitude issues involving quantization are not about quantizing the model, they are about quantizing the cache. See llama.cpp#27109 (prefill drops 20×), #27761 (pp512 drops 68%), and #10552 (−16% with a draft model) in §6.1.
90% of issues mentioning KV cache are memory issues. It is the component that grows linearly with context and that no weights-file-based estimate captures. This is the measured justification for including n_ctx as a mandatory input feature.
The retained set has zero Energy issues and 6 Temperature issues. The initial reading was "the energy axis is not reported." The discard audit showed this reading is wrong: 77 discarded issues mention battery, energy, power, temperature, or throttling — 48 via SCORE_BAIXO, 27 via SEM_MENSURABILIDADE, 2 via exclusion.
The two claims have opposite implications: "not reported" → RQ1 needs to narrow or change source. "Reported, but without a number" → the measurability criterion is what is blind to this axis, and the fix belongs in the filter, not the source. The clearest case is llama.cpp#8760 ("quants IQ1xx/IQ2xx/IQ3xx are overheating, CPU 90C"): it has a measured temperature in the title and falls out for lacking a metric in the prose.
§6The issues found, examples, with the reasoning behind each decision
This section exists so the criterion can be audited concretely, not only in the abstract. Each record carries the issue's claim, what the reporter measured, and why the issue was accepted or rejected. All identifiers are direct links.
6.1 The strongest evaluation instances
These are the issues that, on their own, support the paper's argument. The criterion that sets them apart is not severity, it is having both sides of the deviation measured by the reporter, which makes them directly usable as a test case for the predictor.
tiny/bf16 uses 1.66× the file size; small/bf16 uses 6.7× and is killed by jetsam at 3.80 GB on an iPhone 16.num_ctx), the model no longer fit in 24 GB of VRAM, the scheduler offloaded 10% to the CPU, and generation dropped from 33.8 to 4.7 tok/s.n_predict. The 32,768 default allocates 3,584 MiB, bringing the total to ~7.1 GB.n_ctx."I expect it to use a lot less memory since the models are just 7B and 13B"
#27109: switching the K cache from q8_0 to q4_1 drops prefill from 991–1,276 t/s to 34 t/s (~20×) on an RTX 3090, with generation unchanged.#27761: on ROCm, any KV cache type outside q8_0/q8_0 and q4_0/q4_0 fragments the graph from 2 to 34 splits, pushes ~550–580 MiB to the host, and drops pp512 by 68%. The same build on Vulkan does not have the problem."Right now, users must guess a safe num_ctx. If it's too high, VRAM overflows and Ollama may slow down or fall back to CPU, with no clear feedback."
6.2 The issues that bound the predictor's scope
These three are equally important, and for a different reason: they show what Stargazer v2 cannot predict. The paper is stronger for stating them, because a reviewer who found them alone would conclude we had not looked.
gemma4:26b-mlx-bf16, whose 47.13 GiB peak fits comfortably. Decode is unaffected.(model, hardware, configuration) produces two costs separated by two orders of magnitude, depending only on warm-up state. No static predictor solves this. This is the issue to cite by name when the paper states the limit.6.3 Rejected by consensus: the criterion seen from the negative side
A criterion is only auditable if the discards are also inspectable. All the issues below passed the automatic filter (score ≥ 10) and were rejected by both annotators. The reason column cites the protocol step that failed: Step 1 is "no violated cost expectation"; Step 2 is "no deviation quantified by the reporter".
| Issue | Title | Rejection reason |
|---|---|---|
llama.cpp#12390 | SYCL bug: DeepSeek-V2-Lite-Chat-Q4_K_M does not work as expected | Step 2, the only numbers are in the startup log; "does not work as expected" does not state a cost |
llama.cpp#29082 | [Vulkan] GGML_ABORT cache_k_l0 in Vulkan buffer on SWA models, regression 2.40/2.41 vs 2.39 | Step 1, the expectation is not to abort: a functional regression, not a cost one |
llama.cpp#1240 | QX_4 quantization | Step 1, a proposal for a new quantization scheme, not a violated expectation |
llama.cpp#23335 | Eval bug: draft-mtp changes deterministic output on Qwen3.6 MTP model | Step 1, deterministic output divergence, not cost |
onnxruntime#25472 | std::vector bounds assert on Pi Zero 2 W (arm64, ORT 1.22.0) when running BGE-micro-v2 | Step 2, the 512 MB is the device's spec, not a measurement of consumption |
mlc-llm#2328 | [Bug] InternalError: Vulkan Error, code=-4: VK_ERROR_DEVICE_LOST | Step 2, the only number is 2.00GHz (clock), not a cost measurement |
llama.cpp#19844 | Misc. bug: Llama-3.1-8B-Instruct crash in Vulkan Intel LNL iGPU | Step 1, crash above 4K tokens; functional expectation, no quantified cost |
llama.cpp#13865 | Eval bug: KV cache shifting does not work for Qwen2.5VL | Step 1, an expectation of correctness, not of cost |
The dominant pattern across the 33 rejections is a single one: a functional bug mentioning incidental numbers in a pasted log. This is exactly the false positive the v2 filter was designed to reduce, and it did reduce it from ~83% (v1) to ~55%, but did not eliminate it.
6.4 The six disagreements: and what they revealed about the definition
Six of the 60 issues (10%) received different labels. All went to tiebreak and are resolved. But the relevant finding is not the number: it is that five of the six share the same cause.
| Issue | Ann. 1 | Ann. 2 | Annotator 2's reason for rejecting | Tiebreak |
|---|---|---|---|---|
llama.cpp#932 | 1 | 0 | "4 cores, amd, linux does not identify a reproducible platform" | 1 |
llama.cpp#681 | 1 | 0 | same reason, platform not identified | 1 |
llama.cpp#10552 | 1 | 0 | "no GPU or platform identified, scenario not reproducible" | 1 |
mlx#3625 | 1 | 0 | "no inference model identified, just an operation" | 1 |
mlx#3932 | 1 | 0 | "compilation primitive, no inference model" | 1 |
llama.cpp#21385 | 0 | 1 | (reverse direction) Ann. 2 accepted: "complete triple with measured tok/s" | 0 |
The protocol says: "if the same type of disagreement appears 3 times or more, the operational definition of RELACIONADO is amended". Here the same type appeared 5 times: Annotator 2 requires the triple (model, platform, configuration) to be reproducible; Annotator 1 only requires the deviation to be attributable.
These are two defensible readings of condition 4 of the definition, and the current text does not decide between them. The tiebreak resolved case by case in the direction of attributability ("a vague platform does not void attributability"), but this needs to become a written amendment, dated, before the definitive human round, or the two human annotators will reproduce the same disagreement and the Kappa will measure disagreement over the definition, not over judgment.
This is the most important item to ratify with the advisor, and it is the main value delivered by the pilot round: it found the ambiguity at a cost of zero human hours.
6.5 Suspected false negatives: what the filter likely lost
Outputs from the discard audit, ordered by suspicion score. The decision column is empty: these are candidates, not confirmed false negatives. But reading the titles already indicates the filter errs systematically, not randomly — the pattern is an issue with a clear comparison in the title and a number only inside the log.
| Susp. | Issue | Title | Discard reason |
|---|---|---|---|
| 10 | ollama#17362 | Gemma4 12b MLX generates significantly slower than Gemma4 26b MLX | SEM_MENSURABILIDADE |
| 9 | onnxruntime#32218 | CoreML EP: ~10× inference slowdown on iOS in 1.29 vs 1.28 (ConvNeXt) | SEM_MENSURABILIDADE |
| 9 | llama.cpp#14201 | LLAMA-SERVER is 40% slower than LLAMA-CLI when using identical parameters | SCORE_BAIXO: 9 |
| 9 | llama.cpp#8760 | quants IQ1xx, IQ2xx, IQ3xx are overheating (CPU 90C), Ryzen 9 7950x3d | SEM_MENSURABILIDADE, thermal axis |
| 8 | llama.cpp#27191 | Adreno/Turnip: GPU clock throttling between submissions hurts tg128 | SCORE_BAIXO: 8, thermal axis |
| 8 | mediapipe#6193 | MediaPipeTasksVision Memory Leak after 10 minutes | SEM_MENSURABILIDADE |
| 8 | onnxruntime#27513 | Severe CPU Performance Regression on Intel vs Snapdragon/ARM | SEM_MENSURABILIDADE |
| 7 | llama.cpp#3422 | [User] AMD GPU slower than CPU | SEM_MENSURABILIDADE, score 17, no number in the prose |
| 7 | llama.cpp#5965 | Using OpenCL on Adreno & Mali GPUs is slower than CPU | SEM_MENSURABILIDADE |
| 7 | onnxruntime#10208 | gpu is 4 times slower than cpu on jetson xavier | SEM_MENSURABILIDADE |
Source: data/msr/discard_audit.md, sections 4 and 5. Note that llama.cpp#3422 has a score of 17, well above the cutoff, and is discarded by the hard measurability condition. This is the conscious loss declared in §3.4.
6.6 The 27 confirmed issues, in full
Complete list of the confirmed set, for direct inspection. Those marked with D came from the tiebreak. The category and entry-point columns are Annotator 1's, and they are what produce the corrected distribution in §5.2 (7 Artifact · 20 Toolchain · 0 Platform).
| Issue | Title | Category | Entry point |
|---|---|---|---|
| ggml-org/llama.cpp#29063 | Eval bug: VRAM usage increased from b10680 (SYCL backend) | Memory | Toolchain |
| google-ai-edge/mediapipe#6270 | `RuntimeError: memory access out of bounds` loading gemma-4-E2B-it-web.task on Chrome 146 / macOS / Apple M4 | Memory | Toolchain |
| lmstudio-ai/lmstudio-bug-tracker#2250 | MLX models ignore every user-configured context length, "context auto-fit" always overrides it to the RAM-fitted maximum | Memory | Toolchain |
| ollama/ollama#12342 | Bug: inconsistent to use VRAM and GTT of iGPU of AMD Ryzen AI Processor | Memory | Toolchain |
| microsoft/onnxruntime#20709 | [Mobile] Subgraphs duplicate initializers in RAM during execution | Memory | Toolchain |
| ml-explore/mlx#3658 | Allow force using fused kernel in scaled_dot_product_attention | Memory | Toolchain |
| ollama/ollama#18074 | Granite 4.2 models (8B/3B) do not respect safe default context ( eg. 4096) base on available VRAM and cause OOM kills | Memory | Artifact |
| ggml-org/llama.cpp#27937 | llama-tts: qwen3-tts default context (32768) allocates 3.5 GB of KV cache, 4096 would suffice | Memory | Artifact |
| ggml-org/llama.cpp#21164 | Vulkan regression with 3 GPUS vs 2 GPUS: Qwen3.5 PP drops from ~3500 tok/s to ~350 tok/s vs commit 66d65ec (35B unaffected) | Throughput | Toolchain |
| ollama/ollama#17099 | 0.31.2 regression: vision model (gemma4:31b) memory estimate inflated ~1.2 GiB → no longer fits in VRAM → 7x slower generation (33.8 → 4.7 tok/s) | Memory | Toolchain |
| ollama/ollama#1800 | OOM errors for large context models can be solved by reducing 'num_batch' down from the default of 512 | Memory | Toolchain |
| ggml-org/llama.cpp#932 D | perf: Investigate performance discrepancy with llama-rs - 1.5x-2x slower | Latency | Toolchain |
| google-ai-edge/LiteRT#7787 | Tensor SDK Beta, Dispatcher / runtime protocol mismatch makes AOT path unusable on Pixel 10 Pro XL | Latency | Toolchain |
| ggml-org/whisper.cpp#3088 | Memory Leak in whisper_full without Core ML (master branch) on macOS Apple Silicon? | Memory | Toolchain |
| ggml-org/llama.cpp#23181 | Misc. bug: cudaGraphInstantiate OOM during prompt-cache checkpoint creation under sustained unique-prompt load (Q4_K_M MoE + --n-cpu-moe; --cache-ram 0 does not | Memory | Toolchain |
| pytorch/executorch#22968 | [Vulkan] LLM decode ~12% slower than XNNPACK on Adreno | Throughput | Toolchain |
| ggml-org/llama.cpp#27761 | Misc. bug: ROCm FA: only q8_0/q8_0 and q4_0/q4_0 KV cache works, all other types cause 34 graph splits (pp512 drops ~68%) | Throughput | Artifact |
| ggml-org/llama.cpp#26396 | SYCL: b10216 oneMKL FA path regresses 200K prompt prefill on Arc Pro B70 by ~35% | Throughput | Toolchain |
| ml-explore/mlx#3625 D | [Performance] conv_general 5D (3D convolution) is 2-5x slower than equivalent per-frame 2D conv | Latency | Toolchain |
| ggml-org/llama.cpp#27109 | CUDA: 4-bit KV cache (q4_1/q4_0) collapses prefill to ~34 t/s on qwen35 hybrid (RTX 3090, MMQ guard passes) | Throughput | Artifact |
| microsoft/onnxruntime#29538 | [Performance] Large CPU memory regression starting in 1.24 (Java, macOS arm64) for fully-convolutional models with dynamic input shapes | Memory | Toolchain |
| ml-explore/mlx#3932 D | Memory leak when compiled function contains multi-output op | Memory | Toolchain |
| ollama/ollama#17829 | MLX engine: no prompt/prefix caching between requests (full re-prefill every agent step) | Latency | Toolchain |
| ggml-org/llama.cpp#10552 D | Misc. bug: [server] Using q8_0 for KV cache reduces performance when also using a draft model | Throughput | Artifact |
| mlc-ai/mlc-llm#1827 | [Bug] Speculative decoding results in huge memory usage | Memory | Artifact |
| ggml-org/llama.cpp#681 D | q4_1/f16 model is slow | Latency | Artifact |
| lmstudio-ai/lmstudio-bug-tracker#2074 | Responses API `previous_response_id` chaining re-prefills an exponentially-growing context (10–30× the real conversation), multi-turn far slower than chat-comp | Latency | Toolchain |
27 of 60 sampled issues · filter's measured precision: 45% · generated from data/msr/issues_sample_eval.csv.
§7The literature: what was read and what each paper does for the research
The reading library has 11 papers in 4 groups, all with a versioned PDF in references/ and a README per group explaining each one's role in the text. The organization is not thematic for convenience: each group answers a different objection an ICSE reviewer will raise.
The inference cost of a model on a device (memory, latency, energy, thermal behavior) is only discovered after the model reaches the device, and no release process requires that cost to be estimated beforehand, for a heterogeneous fleet.
The Phase 0 gate found no paper that defines the problem in these terms, but found many that measure or predict inference cost. Hence the reading's three tasks: (a) show the phenomenon is real and quantified; (b) map the estimators closely enough that the gap argument survives a reviewer who knows them; (c) import framing, empirical method, and precedent from software engineering.
Group A: evidence that the cost is real and shows up late
The two measurement studies that demonstrate the phenomenon. They serve as motivation and as a controlled counterpart to the issue corpus: they measure in the lab what the issues report in the field.
Group B: the state of the art in estimation (read adversarially)
These four are read with a fixed question: what exactly does this paper not claim, and can that be said in a sentence a reviewer would accept? CARB and Tu et al. are the two closest neighbors found in Phase 0, that is, the biggest risk to the originality argument.
Group C: framing and method transfer
The paper is a software-engineering paper, and the committee judges process arguments by software-engineering standards. Varghese provides the agenda framing; Singularity provides the worst-case method and the precedent that resource cost is a testable property before release.
Group D: empirical method for the issue corpus
These three are not about inference cost. They are the software-engineering method behind our half of the paper: how a corpus of real faults is mined, organized, and defended. They are read against the protocol, not against the gap argument, and are the direct basis for everything in §3.
tok/s, GGUF, and Q4_K_M in the matrix. The pretest recorded that omitting tok/s cost 100% recall on ggml-org/llama.cpp. The record of rejected terms (§3.2) also follows its replicability requirement.Complementary methodological references (used, outside the PDF library)
| Reference | Where it is used in this study |
|---|---|
| Islam, Nguyen, Pan, Rajan. A Comprehensive Study on Deep Learning Bug Characteristics. ESEC/FSE 2019 | Base protocol for the entire study: vocabulary derivation by symptom (§3.2), sample stratification by repository (§3.5), funnel table format (§4), entry-point taxonomy adapted from pipeline-stage stratification (§3.6) |
| Islam, Pan, Nguyen, Rajan. Repairing Deep Neural Networks: Fix Patterns and Challenges. ICSE 2020 | Supports the fix and regression terms in the keyword matrix; reference for the distinction between progressive and load-time exhaustion |
| Cohen. A Coefficient of Agreement for Nominal Scales. 1960 | Inter-annotator agreement statistic (§5.1) |
| Landis & Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 1977 | Interpretation scale and the Kappa ≥ 0.61 acceptance criterion (§3.5) |
| Byrt, Bishop, Carlin. Bias, prevalence and kappa. J. Clin. Epidemiol., 1993 | PABAK and the prevalence and bias indices, which control for the Kappa paradox (§5.1) |
| Dantas Cruz, Paul, Song, Tilevich. STARGAZER. IEEE SMDS 2020 | The work v2 extends; defines the regression targets the keyword matrix has to cover (§3.2, rule 2) |
| Zou et al. EnerInfer: Energy-Aware On-Device LLM Inference. 2026 | Supports the energy-axis terms (battery, energy, power), which were run and whose negative result is reported in §5.6 |
There still is no paper that defines the problem as a missing release step. This is the Phase 0 gate's finding and the reason the paper exists. None of the 11 papers predicts memory or thermal behavior across a device fleet, and this absence is the cleanest formulation of the contribution space.
Open risk: Kanyiki (AIware 2026, doi:10.1145/3805760.3814901), the abstract has not yet been obtained and the PDF has no known open access. It needs to be read before publishing the gap argument. It is the only known gap in the review.
1. The numbers above came from abstracts and opening pages, not a full read with page-by-page checking. They are recorded in the group READMEs with that mark. None of them should go into the paper's final text without rechecking against the PDF, which applies especially to the claims we will use adversarially (CARB and Tu et al.), where an imprecise citation is fatal.
2. Seven of the 11 papers are 2026 preprints (arXiv, with no completed peer review). This is unavoidable given the topic's time window, but it needs to be stated: the "state of the art" against which the gap is measured is, in large part, unreviewed literature, and a reviewer may legitimately point this out.
§8Threats to validity and declared biases
Structured according to Codabux et al. (2025)'s categories. None of these items is a novelty discovered during review; all were measured during the pipeline and are recorded in the artifacts. The purpose of stating them is for the paper not to be surprised by them.
External validity: who the corpus applies to
Of the 307 issues with a specific platform, 201 (65%) are Apple Silicon. The llama.cpp, ollama, and mlx communities are mostly Mac. Every conclusion about "on-device" is biased toward high-capacity unified memory, the opposite of the constrained Android/edge cohort. Comparison: Android 76 · iOS/iPhone 86 · Jetson 16 · Raspberry Pi 7.
Consequence: this is the study's most serious bias, because the overrepresented cohort is precisely the one that suffers least from memory pressure.
llama.cpp (330) + ollama (124) = 73% of retained issues. Declarable, and not fixable through sampling — the sample is proportional by construction, so it reproduces the imbalance instead of compensating for it. Fixing it would require disproportionate sampling, which would in turn bias the population frequencies.
Forums, Discord, r/LocalLLaMA, and app-store reviews are left out, and that is likely where the energy cost, absent here, is perceived and reported. Reinforced by the grid's empty papel:aplicativo cell (§3.1).
ROCm/ROCm absent
300 queries, HTTP 422, zero issues. The study has 11 repositories, not 12. Cause identified later: renaming to ROCm/legacy-rocm-build (§3.1). Pending: update TARGET_REPOS and recollect the repository.
Construct validity: whether we measure what we claim to measure
Discards legitimate qualitative reports. It is the measured cause of the absence of the energy axis (§5.6) and of discarding llama.cpp#3422 despite its score of 17. This is the criterion to revisit if RQ1 is kept with the energy axis included.
Already corrected and documented in §5.2, but remains a threat because the correction applies to 27 issues and the other 593 remain classified by the refuted heuristic. Any entry-point frequency over the 620 is invalid until labelling is extended.
"Attributable" vs. "reproducible" in condition 4, the cause of 5 of the 6 disagreements (§6.4). Needs a written amendment before the human round.
Internal validity: the temporal asymmetry, and why it is not an artifact
2026 issues are 34.6% of the raw corpus but 60.2% of the filtered one. The first hypothesis, that the filter favors the recent template format, was tested and rejected: the template's expectation section is more common in 2023 (19.3%) than in 2026 (7.4%), and the median score is flat across years.
The measured cause is different: the practice of reporting cost matured.
| Indicator | 2023 | 2024 | 2025 | 2026 |
|---|---|---|---|---|
| Median prose (characters) | 914 | 819 | 982 | 1,896 |
| Numeric metric in the prose | 18.2% | 18.4% | 23.1% | 45.2% |
| Quantization mentioned | 24.7% | 27.3% | 35.0% | 52.1% |
| KV cache / context | 4.4% | 8.0% | 9.6% | 20.4% |
| Regression between versions | 2.5% | 3.7% | 6.2% | 20.2% |
| Specific hardware | 39.3% | 29.3% | 21.2% | 24.4% |
| Filter retention rate | 9.2% | 3.8% | 4.0% | 13.6% |
In 2023 the reporter said "doesn't work on my Mac," which is why specific hardware is more frequent. In 2026 they say "Q4_K_M, 8k of context, 2× slower since build bNNNN, 70 °C."
Frequencies cannot be compared across years. Any observed difference would be a change in reporting practice, not in the problem's incidence. Every longitudinal claim in RQ2/RQ3 needs to state this confound, or be dropped.
Conclusion validity: sample size
The 95% CI for the 26% Artifact figure is ≈ 11%–46%. The structural findings are robust because they do not depend on classification: the backend lift, the median magnitude of 4×, the absence of the energy axis. The per-category frequencies are not, and should be reported with an interval or held back until labelling extends to the 620.
§9State of the deliverables and reproducibility
Done and verified
- Repository selection protocol declared and executed against the API: funnel E0→E6 with 172 approved, context signal, A6 sensitivity, bias report, and blind screening protocol
- Keyword matrix fully traceable: 26 terms, each with a justification and a reference; 5 rejections recorded; API constraints verified
- Real collection with full provenance: 3,600 queries, 7,889 issues, two anomalies handled (truncation bisection, documented HTTP 422)
- Filter v2 with a declared criterion, open weights, cutoff sensitivity analysis, and a frozen, reproducible v1 baseline for comparison
- Stratified sampling with a fixed seed and Kappa milestones as an early-warning control
- Complete funnel generated by script, end to end, including the per-repository table
- Discard audit built, with strata, declared weights, and a recall formula fixed a priori
- Reading library: 11 PDFs in 4 groups, with each paper's role in the written text
Pending, in priority order
- Amend the operational definition, "attributable" vs. "reproducible" (§6.4). Blocks everything downstream, because without it the human Kappa measures disagreement over the definition.
- Definitive human labelling round of the 60 issues, in parallel and independently (Tiago + Isaque/Breno). Replaces the Kappa and every frequency derived from it.
- Fill in the audit of 45 discards → first recall estimate. Today the study reports precision without recall.
- Human round of M1–M3 screening of the 172 approved repositories (E7) → decide whether the list of 11 holds or expands. The LLM pilot included 86, already under collection, but it does not replace this round.
- Update
ROCm/ROCmto the resolved name,ROCm/legacy-rocm-build, inTARGET_REPOS(scripts/msr/common/repos.py). - Recheck every literature number against the PDF before it enters the paper, with priority for CARB and Tu et al.
- Obtain and read Kanyiki (AIware 2026) before publishing the gap argument.
- Recalculate every frequency in
results_analysis.mdagainst the definitive labelled set.
Reproducibility
Every number in this report comes from a script-generated artifact. There are only three hand-written files (keyword_matrix.md, repository_matrix.md, annotation/PROTOCOL.md), and they are pipeline inputs, not outputs. The generated ones carry a "generated by script, do not edit by hand" banner at the top, because editing them would make the paper's number diverge from the dataset's number, which is exactly the problem build_funnel.py exists to prevent.
| Artifact | What it contains |
|---|---|
data/msr/README.md | data status, collection caveats, labelling spreadsheet schema, operational definition, acceptance criteria |
data/msr/results_analysis.md | full analysis of the 620 retained issues: descriptives, RQ1/RQ2/RQ3, magnitude, scope, biases, individual reading of the 12 best issues |
data/msr/funnel_diagram.md | Mermaid funnel, per-repository table, discard anatomy, dataset distribution |
data/msr/kappa_report.md | Kappa per milestone, diagnostics, confusion matrix, agreement per repository, disagreements, tiebreak procedure |
data/msr/filter_v1_v2_comparison.md | why v1 was replaced, with the measurement that motivated each of the three changes |
data/msr/discard_audit.md | discard reasons, suspicion weights, strata, the 30 most suspicious discards, the lost energy axis |
data/msr/top_issues.md | the 40 top-scoring issues, with the reason for each, the page to check whether what the pipeline calls a "good issue" matches the advisor's judgment |
data/msr/repository_selection.md · repository_validation.md | repository funnel, context signal, A6 sensitivity, and retroactive validation of the current list, with the grid's empty cells |
data/msr/selection_bias_report.md | test for concentration in Apple and in app-role repositories, original list against derived list |
data/msr/annotation/SCREENING_PROTOCOL.md | operational definitions of M1–M3, agreement milestone at row 30, and the screening tiebreak procedure |
data/msr/mining_provenance.json | collection provenance: timestamps, queries, and requests per repository, applied API limits |
references/ + 4 group READMEs | 11 PDFs with each paper's role, evidence, and limits |
The complete pipeline, from scratch:
pip install -r scripts/msr/requirements.txt export GITHUB_TOKEN=... python scripts/msr/00_selection/select_repos.py # repository funnel python scripts/msr/00_selection/bias_report.py # bias of the derived list python scripts/msr/00_selection/split_screening.py --limit 30 # blind screening spreadsheets python scripts/msr/01_mining/mine_issues.py --dry-run # validates the query plan python scripts/msr/01_mining/mine_issues.py --fresh # raw collection python scripts/msr/01_mining/mine_issues.py --requery-truncated python scripts/msr/02_filtering/filter_issues.py # criterion v2 (default) python scripts/msr/02_filtering/filter_issues.py --criterion v1 # reproduces the baseline python scripts/msr/03_annotation/sample_for_annotation.py # stratified sample python scripts/msr/03_annotation/split_for_annotators.py # two independent files # … manual labelling by the two annotators … python scripts/msr/03_annotation/merge_annotations.py python scripts/msr/04_agreement/calculate_kappa.py --strict # agreement python scripts/msr/02_filtering/inverse_filter.py --n 45 # discard audit python scripts/msr/05_reporting/build_funnel.py # funnel + tables
Without a token, make_synthetic_dataset.py generates a synthetic dataset to run the pipeline offline. select_repos.py --validate-current runs without a token (public API, 60 req/h).
§10Open questions, what needs deciding with the advisor
In this order, because the first blocks the others.
llama.cpp#681 and #932 (hardware described only as "4 cores, amd, linux") drop out, and the measured precision falls from 45% to ~37%. If it is "attributable," they stay. Both are defensible; what is not defensible is starting the human round without deciding.papel:aplicativo cell in the grid — that is, it is a repository-selection problem, not a filter problem.papel:aplicativo, acelerador:qualcomm), keeping the current 11.ollama#16051 (cold vs. warm, 60–400×) is the counterexample to cite by name, but I want to confirm this does not compromise the framing before writing the contribution section.