Study Overview
Stargazer v2 mines GitHub issue trackers from 86 repositories of on-device LLM
inference engines (llama.cpp, MLC-LLM, ExecuTorch, Ollama, and peers). It collects
issues where developers report observable symptoms of execution cost problems —
out-of-memory crashes, throughput regressions, thermal throttling, quantization
failures — and classifies them by resource category, hardware platform, and entry
point. The study extends Dantas Cruz et al. (IEEE SMDS 2020) from edge cluster
applications to the 2022–2026 generation of on-device LLMs.
Research Questions
RQ 1 · Categories
Which categories of resource problems are reported when running LLMs on-device?
RQ 2 · Combinations
Which model × hardware × backend combinations produce the most problems?
RQ 3 · Quantization
What is the impact of quantization parameters on reported execution cost?
RQ1 · Categories
RQ2 · Combinations
RQ3 · Quantization
→
→
→
resource_category
(model, hardware, repo)
quantization + numeric_metrics
Figure 1 — Each research question maps directly to a column group in the issue-labeling schema.
Study Design & Methodology
1 · Repository Selection Funnel
A repository is included when it satisfies both mandatory criteria:
- M1 — Inference scope: runs or packages LLM inference as a primary function (not a wrapper, tutorial, or demo)
- M2 — Device path: documents a deployment path to the end-user device (mobile, embedded, or consumer laptop without discrete GPU)
- M3 — Issue mass: at least 50 closed issues discussing execution behavior (not only build or CI)
Automatic criteria A1–A6
Pilot screening (M1–M3)
Unique repositories
3,175
Not fork, not archived (A1–A2)
3,075
Issues enabled (A3)
3,029
Active tracker, 12 mo (A5)
246
≥ 50 closed issues (A4)
172
Included in pilot screening (M1–M3)
86
Figure 2 — Repository selection funnel. Bar widths are √n-scaled; the large drop at step 5 reflects the ≥ 200 stars threshold.
Adjudication status
The two pilot screeners agreed with κ = 0.731 on the inclusion decision.
Another 23 candidates have screener disagreements and await adjudication.
Of the 12 repositories from Round 1, 11 pass the current funnel;
the other 75 are new to the study.
2 · Issue Mining Funnel
Round 1 (11 repos)
Round 2 (86 repos)
Retained by automatic filter (v2)
Figure 3 — Round 1 vs. Round 2 at each mining stage. Each group scaled independently; Round 2 added 277 issues via date-bisection recovery and 159 via repository rename fix (filter retention: 8.7 %).
Collection note — Round 2 provenance
Round 2 collection closed 2026-10-08. Two post-collection corrections:
- Date bisection (+277): Five llama.cpp queries hit the 1,000-result API cap and were re-run with date ranges to recover truncated results.
- Repository rename (+159): 303 queries for
mudler/vllm.cpp had been rejected during collection because the repository migrated to localai-org/vllm.cpp; re-run with the new name.
Final count:
12,629 issues from 84 of 86 repositories.
The automatic filter retains
1,094 (8.7 %).
3 · Search Strategy
Queries are built as a cartesian product of two vocabulary axes, plus four standalone primary queries. Every query is scoped to a specific repository (repo: qualifier). Duplicates are removed by (repo, issue_number) pair.
Resource axis — 22 terms
"out of memory"
OOM
"memory leak"
"KV cache"
"context length"
RAM
VRAM
"tokens/s"
"tok/s"
latency
slow
regression
thermal
throttling
throttled
battery
energy
power
quantization
"inference speed"
"slow inference"
"prefill" ★
Device axis — 16 terms
android
ios
iphone
ipad
pixel
snapdragon
NPU
"on-device"
mobile
edge
"Apple Silicon"
Jetson
ARM
raspberry
GGUF
INT4 ★
★ Added 2026-10-10 after 20-result pre-test (≥ 50 % precision threshold).
Grid total: 22 × 16 = 352 queries per repository.
Full visual → keyword map.
In addition to the grid, four standalone primary queries cover the highest-density symptom–device intersections:
is:issue ("out of memory" OR OOM OR "KV cache") AND (android OR snapdragon OR "on-device")
is:issue ("tokens/s" OR "tok/s" OR latency) AND ("Apple Silicon" OR ios OR jetson)
is:issue (thermal OR throttling OR battery) AND (npu OR mobile OR edge)
is:issue "model conversion" AND (GGUF OR ollama OR NPU)
Preliminary Results
Key result — Inter-annotator agreement (Round 1 pilot)
Cohen's Kappa = 0.795 at the 100 % mark of the issue-labeling task —
substantial agreement range (95 % CI: 0.64 – 0.95, Landis & Koch 1977).
This measures agreement between two LLM-operated evaluation roles; it is not
human–human agreement. Acceptance threshold for the human round: κ ≥ 0.61.
Both agreed: include (22)
Disagreements, pending adjudication (6)
Both agreed: exclude (32)
Figure 5 — Pilot sample breakdown across 60 labeled issues (Round 1). Human annotation round not yet conducted.
Limitation — LLM annotation bias
All current agreement numbers (κ = 0.795, confirmed = 22) were produced by two
LLM-operated annotation roles running on the same base model. They cannot substitute
for human inter-annotator agreement and should not be cited as a measure of labeling
reliability. The definitive annotation round with two human annotators has not been done yet.
Status & Next Steps
Stage 00
Select
repos
86 selected
23 pending adjud.
Stage 01
Mine
issues
12,629 collected
84 / 86 repos
Stage 02
Filter
(v2)
1,094 retained (in memory)
outputs pending #17
Stage 03
Sample
& label
Pilot done (LLM)
Human round pending
Stage 04
Agreement
& analysis
Blocked on
Stage 03
Figure 6 — Pipeline stages. Green = complete; amber = in progress / partial; dashed = pending.
-
✓
Stage 00 — Repository screening: 86 repositories selected; 23 disagreements pending adjudication.
partial
-
✓
Stage 01 — Issue mining: 12,629 issues collected from 84 / 86 repositories.
complete
-
◑
Stage 02 — Automatic filter: 1,094 issues retained (8.7 % retention) on the 86-repository corpus, computed in memory by the funnel report. The filtered outputs for that corpus are not persisted yet (
#17), so sampling and annotation still run on the 11-repository corpus.
partial
-
◑
Stage 03 — Annotation: Round 1 pilot labeled (60 issues, LLM). Round 2 sampling and human annotation round not done.
in progress
-
◑
Stage 03-A — LLM pre-screen: two models from different developers label the sampled issues and the candidate repositories with the annotators' protocol text, to order the human round. Implemented and tested; the first run, two local models on both funnels, is committed and its agreement is on the pre-screen page. Its review queue has not been reviewed yet. Never a reportable result. See
LLM Pre-screen.
first run done
-
○
Stage 04 — Agreement & analysis: Human κ not yet measured.
pending
-
◑
Stage 07 — Criteria check (before the human round): stratified sample over kept
and discarded issues, read criterion by criterion by a local LLM (Qwen 2.5 3B on llama.cpp), plus MLOps/LLMOps detection from GitHub Actions, manifests and issue text. Pilot on the 11-repository corpus; to be re-run on the definitive corpus after #17. See
Criteria Validation.
pilot
-
◑
Stage 06 — Exploratory clustering (#8): retained issues clustered by embedding, out-of-scope region identified, five-class taxonomy proposed. Run on the 11-repository corpus; re-run after #17. See
Issue Clusters.
proposal
-
◑
Keyword grid: 22 resource × 16 device terms validated. Gap terms (TTFT, TPOT, SoC, Hexagon, ANE) documented but not yet added.
partial
-
○
Inverse filter evaluation: Blocked on issue #15 (vllm.cpp re-mine by Tiago).
blocked
Key Literature
| Reference | Role in this study |
|---|
| Islam et al. ESEC/FSE 2019 · ICSE 2020 | Symptom-seeded vocabulary protocol; fix-pattern taxonomy |
| Codabux et al. arXiv:2501.01903, 2025 | MSR protocol; 20-result pre-test exclusion rule; jargon adherence |
| Liu et al. arXiv:2506.09713, 2025 | Closest direct precedent — 929 bugs in llama.cpp / MLC-LLM (κ = 0.927) |
| Cai et al. arXiv:2409.16559, 2024 | 994 LLM project issues; separates Memory and Performance categories |
| Chen et al. ICSE 2021 | Methodological baseline — 304 faults in TF Lite and Core ML |
| Cai et al. arXiv:2607.05475, 2026 | NPU bottleneck analysis; validates NPU, snapdragon, ios device axis |
| Tummalapalli et al. arXiv:2603.23640, 2026 | iPhone 16 Pro thermal throttling documentation (40 → 22 tok/s) |
| Dantas Cruz et al. IEEE SMDS 2020 | Stargazer v1 — origin study; defines energy/memory regression targets |
Study Documents (Round 1)
- Repository and issue funnels — selection criteria, screening of 86 repositories, collection results.
- Criteria validation — what the filter keeps and excludes, checked rule by rule by local LLMs on a stratified sample; MLOps/LLMOps usage of projects and reporters (pilot).
- Issue clusters — embedding-based map of the retained issues, labelled precision per theme, proposed taxonomy and recommendation (#8).
- Methodological dossier (PT-BR) — full protocol, design decisions, term rejections.
- MSR study report (PT-BR) — per-repository funnel, filtered corpus analysis, limitations.
Suggested order: methodological dossier → study report. All describe Round 1.
Full bibliography → references.html ·
Keyword matrix → keyword-matrix.html ·
Visual keyword map → keywords-viz.html ·
Repository: luanaa2005/research-analysis-stargazer ·
Updated 2026-10-12