Stargazer v2 · Mining software repositories · status at 2026-10-09 · issue collection finished 2026-10-08 18:14

Repository and issue funnels

Funnel 1 selected 86 repositories in a pilot screening. Funnel 2 has mined their issues: 12,629 collected and 1,094 kept by the automatic filter. Sampling and labelling, which decide which issues become evidence for RQ1–RQ3, come next. In both funnels, each bar is what survives the criterion on its left, and each box on the right is what that criterion removed or added.

Preliminary. The repository screening (E7) and the issue labels on this page come from a pilot run by two LLM-operated rater roles, not from the human round, and may not be cited in the paper. In funnel 2, collection and filtering are measured; sampling, κ and labels have not run yet, so those hatched numbers are projections.

Funnel 1 · repositories3,430 → 86selected by A1–A6, then pilot screening M1–M2
Screening agreementκ 0.731inclusion over 172 candidates; LLM pilot
Funnel 2 · issues12,629 → 1,094raw issues → kept by the v2 filter (8.7%)
Mining · finished100%26,058 queries in 19 h 17 min · 84 of 86 repositories returned issues
Measuredcounted from the data LLM pilottwo LLM-operated rater roles; a human round replaces it Projectedestimated from the two earlier labelling rounds Pendingdefined but not executed yet
The whole pipeline

Six stages, two funnels, 86 repositories mined

FUNNEL 1 repositories FUNNEL 2 issues · mined 2026-10-08 00_selection Select repos 86 selected · LLM pilot repo_screening.csv 01_mining Mine issues 12,629 raw · 84 of 86 repos issues_raw.jsonl 02_filtering Filter (v2) 1,094 kept · 8.7% of raw issues_filtered.jsonl 03_annotation Sample & label 60 next · by repository issues_sample_eval.csv 04_agreement Agreement κ 0.6–0.8 expected range kappa_report.md 05_reporting Report ≈ 24 RELATED, projected funnel_table.md 86 repos 2 screeners LLM pilot M1–M3 keyword_matrix.md one vocabulary, both funnels context signal queries Discard audit after filtering inverse filter 2 raters · LLM pilot run after sampling blind labels
Stages are the folders of scripts/msr/. The hand-off from funnel 1 to funnel 2 has happened: the 86 selected repositories were mined in an isolated run, kept apart from the committed 11-repository corpus in data/msr/. Solid boxes are measured, the dashed blue box is a projection, and dashed amber boxes are the next steps.
Funnel 1 · stage 00_selection

Repository selection: automatic gates A1–A6, then blind screening M1–M3

Sources: repository_selection.md (2026-10-05) for A1–A6 and the two pilot screening sheets for E7 (local dry-run files, not committed). The gates run in the order the script applies them; A5 and A4 cost API calls, so they run last, on the fewest candidates.

CriterionRepositories that surviveRemoved at this step
  1. SOURCE
    12 seed queries
    GET /search/repositories over name, description and topics. Each query targets a declared axis.
    llm inferenceon-device inferencemobile llmedge ai inferencegguflocal llm appoffline chat assistant llmnpu inferencequantization inference enginetopic:llm topic:androidtopic:on-device-aitopic:edge-computing topic:machine-learning
  2. E0
    Raw search results
    Summed over the 12 queries. A repository found by two queries counts twice here.
    3,430
  3. E1
    Unique repositories
    Deduplicated by full name.
    3,175
    −255duplicates across queries
  4. E2A1A2
    Not a fork, not archived
    Forks duplicate the parent's history; an archived tracker can't receive reports (Kalliamvakou 2014; Munaiah 2017).
    3,075
    −100forks or archived
  5. E3A3
    Issue tracker enabled
    The unit of observation is the issue.
    3,029
    −46issues disabled
  6. E4A6
    At least 200 stars
    Social-attention floor and the protocol's most contestable gate (Borges & Valente 2018). A repository that is the only one in a coverage cell may re-enter by exception.
    508
    −2,521under 200 stars; 0 re-entered by the coverage-cell exception
  7. E5A5
    Tracker active in the last 12 months
    At least 20 issues updated in the window. Measures the tracker, not the code.
    246
    −262dormant tracker
  8. E6A4
    At least 50 closed issues
    Evidence of a sustained engineering process. Last automatic gate.
    172 candidates
    −74too few closed issues
  9. Context signal · not a gate
    Asks what A1–A6 never ask: is the repository about on-device cost? Uses the same vocabulary as funnel 2.
    Tracker has ≥ 1 on-device cost issue (probe queries Q1–Q3)115 / 172
    Own text names a resource term and a device term5 / 172
    0removed. It only orders the screening sheet: tracker probe, then text score, then issue mass.
  10. Kappa sampling · repositories
    Two screeners fill separate copies of the 172-row sheet, top-down, without seeing each other's. They stop at row 30 and compare before going on.
    LLM pilot
    Cohen's κM1 inferenceM2 device pathM3 cost reportsInclusion
    Checkpoint, rows 1–301.000.670.470.67
    Full sheet, 172 rows0.7170.8270.7790.731
    Checkpoint passedM1 and M2 both ≥ 0.61 at row 30, so screening continued to row 172. Inclusion agreement over the full sheet: 86.6%.
  11. E7M1M2M3
    Manual screening
    Include only if M1 and M2. M1: its code runs or packages LLM/ASR inference. M2: its own docs show a path to an end-user device. M3 (tracker gets cost reports) is recorded as a caveat and never excludes.
    86both screeners include
    23disagree → adjudication
    63both screeners exclude
  12. SELECTED
    Selected corpus
    11 of the study's original 12 repositories are in it. The other 75 are new to the study.
    86
    +0 to 23after adjudication. Not adjudicated yet, so the run uses the 86.
  13. OUTPUT
    Hand-off to funnel 2
    The 86 names are passed to the miner with --repos.
    86 repositories → mine_issues.py --mode both → 12,629 issues
Bar width is proportional to log₁₀ of the count, with 10 at zero width, so the drop at A6 (3,029 → 508, a factor of six) reads as a step rather than a cliff. The E7 split of the 172 candidates is drawn linearly.

How much the 200-star threshold decides

Stars ≥Pass A6Pass A1–A6Original 12 kept
10066118412
20050817211
50031515111

A fivefold change in the threshold (100 → 500 stars) moves the approved pool by 33 repositories (184 → 151). Of the original 12, only lmstudio-bug-tracker depends on where the line sits.

The original 12 repositories, through the whole funnel

11 of 12 make it through E7. lmstudio-ai/lmstudio-bug-tracker fails A6, so it never reaches screening, although it supplied 297 raw issues to the earlier corpus.

ROCm/ROCm passes, under the name GitHub now resolves: ROCm/legacy-rocm-build. The old name returned HTTP 422 to every issue query. The new name yielded 304 issues in the 86-repository run.

Funnel 2 collection finished

The 86 selected repositories were mined from 2026-10-07 22:57 to 2026-10-08 18:14 (19 h 17 min): 26,058 queries, 26,210 requests and 633 rate-limit waits, with no crash and every network retry recovered. The run produced 12,193 unique issues. Five llama.cpp queries had hit the API's 1,000-result cap; on 2026-10-09 they were re-collected by date bisection (118 requests, no window left over the cap), adding 277 issues. The same day, the 303 queries of localai-org/vllm.cpp, rejected during the run because the repository had been renamed, were re-run under its new name, adding 159 issues. The corpus now holds 12,629 unique issues from 84 of the 86 repositories, created between 2016-08-17 and 2026-10-08. Mining and filtering below are measured; sampling and labelling have not started.

Funnel 2 · stages 01 to 05 · 86 selected repositories

Issues: mine, filter, sample, double-label

The same design as the earlier 12-repository round: Q1–Q3 plus the full keyword grid, the v2 filter, a 60-issue stratified sample and two blind raters. Mining and filtering are measured. Sampling and labelling have not run yet, so the hatched rows after the filter are projections.

CriterionIssues that surviveRemoved or added at this step
  1. SOURCE
    Keyword matrix → issue search
    The API allows 5 boolean operators per query, so the grid expands the matrix into atomic queries: 3 primary plus 300 grid queries per repository.
    Resource (20) out of memory · OOM · memory leak · KV cache · context length · RAM · VRAM · tokens/s · tok/s · latency · slow · regression · thermal · throttl · battery · energy · power · quantization · inference speed · slow inference
    Device (15) android · ios · iphone · ipad · pixel · snapdragon · NPU · on-device · mobile · edge · Apple Silicon · Jetson · ARM · raspberry · GGUF
    26,058 queries86 repos × (3 + 20 × 15), all executedthrottl returned nothing in all 1,290 of its queries (thermal: 178 with results). GitHub search does not prefix-match, so the grid ran on 19 working resource terms.
  2. Q1–Q3
    Primary sweep
    Three 5-operator queries: memory × mobile, speed × Apple/Jetson, thermal × NPU/edge. Run on 2026-10-06; 71 of the 86 repositories had hits.
    1,546
  3. GRID
    Full keyword grid
    Every resource × device pair, deduplicated by (repository, issue number). Measured on the final checkpoint.
    12,629 raw corpus
    +11,083new issues the primary sweep missedIncludes +277 from 5 llama.cpp queries that hit the 1,000-result cap (1,016 to 1,670 reported) and were re-collected by date bisection on 2026-10-09; the rest of their missing matches were already in the corpus via other grid queries. Includes +159 from localai-org/vllm.cpp, formerly mudler/vllm.cpp: renamed during the run, so its 303 queries were rejected (HTTP 422); re-run under the new name on 2026-10-09, they took it from 11 issues (primary sweep only) to 170.
  4. F·SCOPE
    Not datacenter-only
    The v2 filter's only hard exclusion: A100/H100, DGX, EC2, SageMaker, Kubernetes, tensor-parallel, multi-node, unless an edge device is also named.
    12,277
    −352datacenter cohort
  5. F·MEASURE
    A measured number in the prose
    At least one number with a unit (MB, GB, tok/s, ms, °C, mAh, W, %) written by the reporter, outside code blocks, <details> and pasted logs.
    5,451
    −6,826no measured number outside the logs
  6. F·SCORE
    Violated-expectation score ≥ 10
    +3 symptom in title, expected-behaviour section, theory vs. observed. +2 explicit expectation, version regression, unexpected ratio, backend slower than CPU, scales with context, specific hardware, 2+ numbers in prose. +1 comparison, model, quantization, ≥ 3 comments, ≥ 2 reactions, linked fix. Penalties: correctness bug −6 (title) or −4, build error −5, question or feature request −3, prose under 300 chars −3, consumer desktop GPU −2.
    1,094 8.7% of raw
    −4,357score below 10. All 11,535 discards then go to the discard audit (45 sampled by suspicion stratum).
  7. SAMPLE
    Stratified by repository
    Proportional to each repository's share of the filtered set, row order shuffled with fixed seed 2028. With more repositories than slots, small ones get 0–2 issues.
    60
    1,034not sampled (reserve)
  8. Kappa sampling · issues
    Two raters label each of the 60 issues RELATED or NOT RELATED, blind to each other. Cohen's κ is computed on prefixes of the shuffled sheet (5, 10, 20, 50 and 100%). Prefixes under 20 rows are reported but don't decide anything.
    Next step
    1.0 0.8 0.6 0.4 0.2 0.0 target 0.61 0.603 0.795 5% 10% 20% 50% 100% n=3 n=6 n=12 n=30 n=60
    12 repos, full grid · κ 0.795 [0.64, 0.95] 86 repos, primary sweep only · κ 0.603 [0.40, 0.80]
    Both are earlier LLM rounds, shown as the reference for this one. Hollow points have n < 20.
    κ ≈ 0.60–0.80Expected range, bracketed by the two earlier rounds. The run reports κ from before adjudication.
  9. LABEL
    RELATED needs all four
    A violated cost expectation · a quantified deviation · execution on the device · a cause that can be attributed (model, configuration or platform). The projection averages the two earlier rounds (22 / 6 / 32 and 27 / 12 / 21).
    ≈ 24RELATED, both raters
    ≈ 9disagree → adjudication
    ≈ 27NOT RELATED, both raters
  10. DATASET
    Evidence for RQ1–RQ3
    Consensus RELATED issues: evaluation instances for the Stargazer v2 cost predictor.
    ≈ 24
    +0 to ≈ 9after adjudication of the disagreements (range 22–27 before it)
Bar width is proportional to log₁₀ of the count, with 10 at zero width. The three filter rows are listed in the order the pipeline uses to assign each discard its reason (scope, then measurability, then score), so the three removals add up to the total discarded. The split of the 60 is drawn linearly. The filter counts come from applying filter v2 to the final issues_raw.jsonl.

How the run ended

RepositoriesCountRaw issuesKept by v2Kept %
From the original 12117,9736177.7
New to the study754,65647710.2
Total8612,6291,0948.7

84 repositories returned issues; crosswk/SayIt and lukaszliniewicz/Pandrator returned none. 54 have at least one issue past the filter. The projections held: at 14:08 the page expected ≈ 11,700 raw (range 11,400–12,300) and ≈ 1,020 kept; the run ended at 12,193 and 1,075, before date bisection added 277 raw and 11 kept issues and the localai-org/vllm.cpp re-query added 159 raw and 8 kept.

Earlier corpus vs. this run

12 ad-hoc repos86 selected repos
Queries3,60026,058 + 303
Raw issues7,88912,629
Kept by v2 filter620 (7.9%)1,094 (8.7%)
Sampled6060
κ (LLM pilot)0.795≈ 0.60–0.80
RELATED / disagree / not22 / 6 / 32≈ 24 / 9 / 27

The 75 new repositories retain more of their issues than the original 11 (10.2% vs. 7.7%) and supply most of the growth over the earlier corpus. The original 11 still hold 63% of the raw corpus and 56% of the kept set.

All 86 repositories: issues per repository
RepositoryRaw issuesKept by v2Kept %
ggml-org/llama.cpp3,9063428.8
ollama/ollama1,7631297.3
unslothai/unsloth639396.1
pytorch/executorch435184.1
mlc-ai/mlc-llm4344410.1
microsoft/onnxruntime380307.9
oobabooga/textgen37092.4
google-ai-edge/mediapipe32551.5
ROCm/legacy-rocm-build304134.3
lablup/mlxcel2937425.3
jundot/omlx2918930.6
kvcache-ai/ktransformers236114.7
lemonade-sdk/lemonade180147.8
localai-org/vllm.cpp170105.9
Blaizzy/mlx-vlm1432114.7
ggml-org/whisper.cpp13453.7
city96/ComfyUI-GGUF13300.0
google-ai-edge/LiteRT-LM1321612.1
mozilla-ai/llamafile12843.1
alibaba/MNN12075.8
xorbitsai/inference12032.5
ml-explore/mlx1112018.0
google-ai-edge/LiteRT10598.6
raullenchai/Rapid-MLX1043836.5
qualcomm/GenieX881112.5
maziyarpanahi/openmed8733.4
apple/coremltools7622.6
gufo-org/gufo7168.5
Samsung/ONE6900.0
containers/ramalama6700.0
microsoft/BitNet6611.5
NVIDIA/TensorRT-Edge-LLM631828.6
DenisovAV/flutter_edge_ai58813.8
ddalcu/mlx-serve552036.4
kkokosa/dotLLM4936.1
withcatai/node-llama-cpp491122.4
warpfront/hipfire4748.5
UbiquitousLearning/mllm4312.3
off-grid-ai/OGAM4137.3
waybarrios/vllm-mlx411434.1
RunanywhereAI/runanywhere-sdks40512.5
Luce-Org/lucebox37410.8
jegly/Box3213.1
software-mansion/react-native-executorch31412.9
1038lab/ComfyUI-QwenVL3000.0
uxlfoundation/oneDNN3026.7
xororz/local-dream3000.0
Mobile-Artificial-Intelligence/maid2800.0
cactus-compute/cactus28517.9
jjang-ai/vmlx27518.5
Michael-A-Kuykendall/shimmy2500.0
argmaxinc/argmax-oss-swift2528.0
OpenNMT/CTranslate22200.0
mlc-ai/web-llm2214.5
turboderp-org/exllamav32229.1
mybigday/llama.rn2000.0
ngxson/wllama2000.0
tetherto/qvac19210.5
PurpleDoubleD/locally-uncensored1715.9
deeeed/audiolab1715.9
Muesli-HQ/muesli1600.0
nntrainer/nntrainer1500.0
shubham0204/SmolChat-Android1516.7
lyogavin/airllm1300.0
nobodywho-ooo/nobodywho1300.0
AtomicBot-ai/Atomic-Chat1218.3
Nehanth/pooled1200.0
drumih/turbo-fieldfare1100.0
jjang-ai/mlxstudio900.0
gokayfem/ComfyUI_VLM_nodes700.0
hybridgroup/yzma7114.3
weicj/vLLM-2080Ti-Definitive700.0
woheller69/whisperIME700.0
Capsize-Games/airunner600.0
Trans-N-ai/swama5120.0
sarim2000/pennywiseai-tracker500.0
AtomicBot-ai/atomic-agent400.0
abdulmominsakib/localmind400.0
pnnbao97/VieNeu-TTS300.0
techjarves/Portable-Local-Studio300.0
Eric-Terminal/ETOS-LLM-Studio200.0
callstackincubator/ai200.0
mindspore-ai/mindspore200.0
QwenAudio/Fun-ASR100.0
crosswk/SayIt00—
lukaszliniewicz/Pandrator00—