Criteria Validation — Stargazer v2 MSR

Scope of this run This validation runs on the 11-repository corpus (— raw issues, — retained by filter v2). The 86-repository collection has only its provenance on disk; its raw issues have not been filtered yet (#17). This run is a pilot. The final criteria validation will be run again on the definitive corpus with the same scripts and seed before any number from it is quoted.

What this stage is, and what it is not Stage 07 validates the criteria, not the issues. It has three jobs: (a) show what the filter excludes, by reading the discarded strata; (b) find which filter rule diverges from the protocol, criterion by criterion; (c) find out whether projects and reporters use MLOps or LLMOps. Classifying the issues with several LLMs and measuring κ between models is a separate piece of work: Tiago's #6. The local reader here does not replace it and is not compared with it as a result.

1. Why this step exists

Stargazer v2 aims to estimate the execution cost of on-device LLM inference without running the model. The MSR study supplies the motivation and the evaluation instances: issues where practitioners expected one cost and observed another. Everything downstream (the Kappa round, RQ1–RQ3 frequencies, the predictor's test set) inherits whatever the filter lets through. So the advisor's request (2026-10-12) was to “take a sample to verify before proceeding and send it to the AI, so we understand what is coming in and define what is being excluded”.

Up to now the pipeline could only measure precision: the annotation round only ever sees issues the filter kept. This step reads the discarded side too, and it checks each rule separately. If the filter and a reader disagree on one criterion, the protocol can be fixed at that rule.

The precedent is Chen et al., ICSE 2021, a study of deployment faults in mobile deep-learning apps (TF Lite, Core ML), mined from GitHub and Stack Overflow. They refined their dataset by manual reading before labelling, ran a pilot on 50% of the posts, and reported κ = 0.819 / 0.743. Two of their findings bear on our motivation: memory and speed issues are 25.5% of the faults in their inference stage, and 20.1% of deployment faults (slow inference among them) do not crash, so developers only find them by experience. That is the gap a cost predictor fills.

What each criterion is for

Criterion (PROTOCOL.md)	Why the predictor needs it	How filter v2 approximates it
C0 reports an execution cost	The predictor's outputs are memory, latency, throughput, energy, temperature.	Resource-symptom regex over the prose.
C1 a cost expectation was violated	Evidence that cost is unpredictable today, which is the motivation itself. Each such issue contains an expected and an observed value.	Expectation section, “expected / should”, regression, ratio, accelerator-slower-than-CPU signals (weighted score).
C2 quantified by the reporter	An evaluation instance needs a number to compare a prediction against.	A number with a unit in the prose, after logs are removed (hard requirement).
C3 on-device execution	Scope: personal devices, not datacenters.	Datacenter regex without any device cohort (the only hard exclusion).
C4 attributable to model / config / device	These are the predictor's inputs (model × quantization × hardware × backend, RQ2/RQ3).	No rule of its own. Model, quantization and hardware are extracted and scored, never required.

2. Sample and reader

The filter is re-applied in memory with the thresholds recorded in filter_stats.json. Its output reproduces stage 02 exactly, and stage 02's files are not rewritten. Each region of its decision space is one stratum. Stratum sizes are kept, so shares can be re-weighted to the population.

Stratum	Definition	Population	Sampled

Each issue goes to a local model served by llama-server (llama.cpp) on a laptop CPU, with no GPU and nothing sent to a cloud API. The model sees the repository, labels, title and the reporter's prose, with logs and code blocks removed (the same prose filter v2 scores). It answers each criterion separately. A JSON schema enforced by the server's grammar sampler guarantees every reply parses. The RELATED verdict is recomputed in code as the conjunction of its answers, exactly as the protocol defines it.

Reader	Model file	Answered	s / issue	Prompt

One local reader is enough for this stage's purpose. It is an instrument to find which rule diverges and what the discarded strata contain, not a classifier of the corpus.

3. What is coming in, and what is being excluded

Share of each stratum that a reader judges RELATED, with 95% Wilson intervals. Kept strata estimate precision. Discarded strata estimate what the filter loses.

Dots: point estimate. Bars: 95% Wilson interval. About ten issues per stratum, so read the intervals, not the dots.

What the discarded strata are made of (issue type, per reader)

4. Criterion by criterion: filter rule vs. reader

For each criterion, the filter's rule is projected onto the same question the reader answered. A disagreement in one direction only points to a rule that is too loose or too strict. A low κ with high agreement is the prevalence paradox, which is why PABAK is shown as well.

Criterion	n	Agreement	κ	PABAK	filter yes · reader no	filter no · reader yes

Before any rule changes The C2 and C0 rows that disagree in only one direction are listed for a person to check in data/msr/criteria_check/review_c2_c0.csv (verdict: reader_right / filter_right / unclear). The reader is a 3B model, so it can be the one that is wrong; a rule is tightened only if the review backs it.

5. Which queries and keywords bring the noise?

Each issue records the mining queries it matched (matched_queries). Each query is split into its keyword terms. For every term: how many issues it collected, how many filter v2 kept (whole corpus, stage 02's decision), and the reader's RELATED share among sampled issues carrying that term, re-weighted by stratum size. A term with high volume and low retention mostly collects issues the study does not use. Volume is concentrated in the keyword grid's GGUF and ARM device anchors.

Term	Issues matched	Retained	Retention	Sampled	Reader RELATED (weighted)

Largest query cores by volume

Query core	Issues matched	Retained	Retention

An issue matched by several queries counts under each of their terms. The sampled column is small per term, so treat it as a pointer only.

6. Do the people and projects behind the issues use MLOps or LLMOps?

The advisor asked whether this can be identified from tools or from Actions. Both can, and they answer for different people:

The project. Its GitHub Actions workflows and root dependency manifests show what it automates: does CI fetch a real model, measure cost, or evaluate quality, and on which hardware?
The reporter. Their issue shows what they run, from tool names in the prose and in pasted tracebacks (LangChain, Open WebUI, LiteLLM, Docker, MLflow, …). The local reader also classifies each sampled reporter's usage_context, which serves as a check on the regex.

Vocabulary: scripts/msr/common/ops_stack.py, one declared list shared by both scans. Ambiguous words found in this corpus are written defensively. Examples: “weights and biases” (network parameters), “evidently”, “lots of airflow” (case fans) and Ollama's own “model registry”. A match means a tool is mentioned, not that it caused the problem.

Repository level: the 12 target repositories (11 yielded issues)

Repository	Workflows	Model in CI	Cost measured in CI	Quality eval in CI	CI hardware	Tools shipped	Profile

Issue level: what reporters run

Tool	Category	Issues (corpus)	Issues (retained)

7. Browse the sample

Every sampled issue, with the filter's decision and the reader's answer. Titles, summaries and reasons are shown as data. The reader's text is model output, not a label.

Id	Issue	Filter	Reader

8. How to read this, and what it does not show

Pilot. The reader is a 3B-parameter model at 4-bit on a laptop CPU. It is a cheap second reader that tells humans where to look. The human round (#7) stays the ground truth (CLAUDE.md rules 2 and 8).
Small strata. About ten issues each. A stratum's interval spans tens of points, so re-weighted totals are an order of magnitude, not an estimate for the paper.
The “labelled pilot” anchor was itself labelled by an LLM pilot (Claude, two written rules), not by humans yet.
Corpus. 11 repositories, not the definitive 86. The run will be repeated on the definitive corpus (see the callout above).
What it does show: which rules the filter and an independent reader disagree on, what each discarded stratum is made of, and which discarded issues deserve a human look before the round starts.

Reproduce

python scripts/msr/07_criteria_check/sample_criteria_check.py            # seed 20261012
llama-server -m qwen2.5-3b-instruct-q4_k_m.gguf -c 8192 --port 8088
python scripts/msr/07_criteria_check/llm_criteria_check.py --label qwen2.5-3b
python scripts/msr/07_criteria_check/ops_stack_scan.py issues
python scripts/msr/07_criteria_check/ops_stack_scan.py repos --repo-set both   # needs GITHUB_TOKEN
python scripts/msr/07_criteria_check/ops_stack_scan.py report
python scripts/msr/07_criteria_check/report_criteria_check.py

Generated reports: data/msr/criteria_check/criteria_report.md, data/msr/criteria_check/ops_stack_report.md. Page data: docs/data/criteria_check.json.

Stargazer v2 · MSR study · Stage 07 criteria check · every number on this page is read from docs/data/criteria_check.json · luanaa2005/research-analysis-stargazer