MSR study · Collection · Between the filter and the human round

Two models read every item before a person does

Two language models from different developers label the same issues and the same candidate repositories, with the same prompt and the same protocol text. Where they agree and where they split orders the human annotation round; their labels are never ground truth and never a result of the study.

First run: committed Pre-screen · not a reportable result Issues and repositories

Results at a glance

How to read κ on this page

Two raters agree part of the time by chance alone. Cohen's κ keeps only the agreement beyond chance: 0 is what chance gives, 1 is perfect agreement. Each κ below is a dot on this scale.

Pre-screen · not a reportable result

These values say how consistently two models apply the written protocol, not whether they are right. They order the human work and point at protocol text to amend; the human round is the reference.

What the models judge

Both models receive the annotators' own protocol, word for word, and answer one item per call in a closed JSON schema. In short:

Funnel 1 · Issues

RELATED only when all four conditions hold

  1. A cost expectation was violated. The reporter expected some memory use, speed or battery life and saw something else: stated, derived, or a regression.
  2. The deviation is quantified. A number with a unit, written by the reporter: MB, tok/s, ms, °C, mAh, W, %. A third-party benchmark does not count.
  3. It runs on the device. Phone, embedded board, laptop or consumer desktop; a datacenter does not count.
  4. It is attributable to a model, a configuration (quantization, context, backend) or an identifiable platform.
One condition missing → NOT_RELATED, for example
"It's slow", no number Crash log that shows MiB Optimization request, nothing violated OOM on an A100 cluster
Decided in advance as RELATED
Regression between versions, with a number NPU or GPU slower than CPU, with a number Feature request born of unpredictable cost
If RELATED, two labels (the RQ1 labels)
resource_category
The cost axis. MemoryLatencyEnergyTemperatureThroughputOther
entry_point
Where the cost originates: the model, the runtime, or the device. ArtifactToolchainPlatform
Funnel 2 · Repositories

Included when M1 and M2 both pass

M1∧M2→include

M3 is recorded as a caveat and never excludes.

M1Runs or packages inference

Its own code is on the execution path: it loads a model and produces tokens or transcripts, or compiles, packages or serves something that does.

C++ runtime for GGUF AOT compiler to a mobile binary Model manager that launches a runtime Vendor SDK with an NPU delegate
Chat UI over a hosted API Weights only Docs, benchmarks, awesome-lists
M2Its own docs show an end-user device

The documentation, not the possibility: nearly anything compiles for ARM in principle.

Android or iOS build steps macOS or Windows install Raspberry Pi or Jetson tutorial Wasm in the browser
Helm chart only "Any CUDA GPU", no consumer device
M3Caveat: the tracker receives cost reports

Reporters describe memory, latency, energy or thermal cost they observed themselves. The study's query hit count is shown to the model as a starting point to check, not as the answer.

What each model saw, and never saw

Sees

An issue

  • Repository, number, title and labels
  • The whole body, never cut

A repository

  • Name, description and topics
  • The study's query hit count for its tracker
  • A neutral sample of its issue titles: the most commented and the most recent
  • The README up to a fixed cap, with the cut stated in the prompt
Never sees
  • The suggested_* columns of the annotation sheet
  • The filter score, or any field the filter extracted
  • The study's cost queries themselves, which would answer M3 for the model
  • The other model's answer
  • Any human label

A model that sees the filter's guess tends to agree with it, and then its agreement says nothing. Evidence is built from an allow-list of raw fields.

The exact criteria text both models received: issues 

  

  

    The exact criteria text both models received: repositories
    

  

  
Prompt templates in scripts/msr/03_annotation/prompts/; answer schemas in common/prescreen.py; the criteria text above is frozen with the run.


  
  
Issues: how the two models voted

  

  


  
Examples, with each model's justification

  


  
The two RQ1 labels, on the issues both called RELATED

  


  

    All metrics and the confusion matrix
    

  


  
  
Repositories: criterion by criterion

  

  


  
Examples, with each model's justification

  


  

    All metrics and the confusion matrix
    

  


  
  
Is each model consistent with itself?

  
Each model rated the first items of each funnel a second time, with a different seed. A model that agrees with
    itself less than with the other model is noisier than the disagreement it takes part in.

  


  
  
What goes to a person

  

  
Splits show where the protocol is read in two ways; a sample of agreements checks that the models are not wrong
    together. Both wait in <funnel>/review_queue.csv.

  

    Who may read the review queue
    
The queue and the run folder show the models' labels. Anyone who will label the same items in the human round must
      not see them before that round closes, or the model answers anchor the ground truth. The examples on this page are
      public: an item shown here can no longer be labelled blind.

  


  
How the pre-screen works


  
  
How it runs


  
A run has three steps and lives in its own folder, data/msr/llm_prescreen/<run_id>/. The evidence and the
    prompt are frozen first; each model then rates the frozen items on its own; the agreement step compares the two
    sheets and writes the queue a person reviews.


  
  
Figure 1 — The two raters never see each other's answers; they meet only in the agreement step.


  

    

      
StepScriptWrites under <run_id>/<funnel>/

      
        
1 · Freeze evidence03_annotation/llm_evidence.py
          evidence.jsonl, the frozen prompt_template.md and protocol_section.md, a block of the run's manifest.json

        
2 · Rate, once per model03_annotation/llm_rate.py
          responses_<rater>.jsonl (the raw answers, which are also the cache), rater_<rater>.csv, retest_<rater>.csv

        
3 · Agreement04_agreement/llm_agreement.py
          sample_eval.csv, kappa_report.md, disagreements.csv, review_queue.csv, and one llm_agreement_report.md for the whole run

      
    

  


  

    Where it sits in the pipeline
    
The pre-screen runs after the automatic filter (stage 02) and before the human annotation round (stage 03).
      The filter score and the suggested_* columns already order work for people; the pre-screen adds a second
      ordering that reads the protocol itself. Once the human round exists, the same tooling compares each model with a
      human sheet and measures how far it can be trusted.

  


  
  
The raters


  
Independence is the reason for using two models. Two fine-tunes of one base model are not independent raters, and
    neither is a distilled model and its base: a DeepSeek-R1 distill is a Qwen or a Llama underneath. The two raters must
    come from different developers and training pipelines.


  

    

      
ProviderFlagHow it is used

      
        
Local open-weight models through Ollama--provider ollama
          Free. One model at a time; the digest of the weights is recorded with each answer.

        
Anthropic API--provider anthropic
          Structured output enforced by the API. Key read from ANTHROPIC_API_KEY, never written anywhere.

        
Gemini API--provider gemini
          Structured output enforced by the API. Key read from GEMINI_API_KEY.

        
Simulated rater--provider fake
          A labeller with a known error rate, for tests and dry runs. Its reports carry a SYNTHETIC banner.

      
    

  


  

    First configuration
    
gpt-oss:20b (OpenAI) with think=medium and qwen3:8b (Alibaba) with
      think=true, both free and local through Ollama: two developers, so two independent raters, at no
      monetary cost.

  


  
No temperature, so stability is measured

  
Current Claude models reject a temperature parameter, so no provider sets one and every model samples with its
    publisher's defaults. Local runs add a fixed seed per replicate: the run reproduces on the same machine, and the retest
    is still a genuine second draw. Each model re-rates a subset of the items and is compared with itself, so stability is
    a number in the report rather than an assumption. No fallback model is enabled either: a refusal rescued by a
    different model would change who the rater is.


  
  
Reading the agreement


  
The binary decisions go through the plumbing of the human round unchanged:
    merge_annotations.py, calculate_kappa.py --prescreen and split_screening.py --merge
    run on the two model sheets as they would on two annotators' files. The report therefore carries the same observed
    agreement, prevalence and bias indices and PABAK as the human Kappa, and is read with the same rules.
    llm_agreement.py adds what two models need and two people do not:


  

    

      
MeasureWhy

      
        
Kappa over the declared classes of resource_category and entry_point, on items both models call RELATED
          These are the RQ1 labels. PABAK is generalised to (k·p₀ − 1)/(k − 1) with the declared number of classes, not the observed one.

        
Kappa per repository criterion: M1, M2, inclusion, M3
          Locates the criterion the models read differently. The human screening merge reports the same four values.

        
Retest Kappa, each model against itself
          The sampling cannot be fixed, so stability is measured. A model that agrees with itself less than with the other model bounds the inter-model value by noise.

        
review_queue.csv
          Every disagreement and invalid answer, plus a seeded sample of agreements: two models can be wrong together.

      
    

  


  

    How to read it
    
  


  
  
Why a local run takes hours


  
With two free local models on a laptop, the first run takes hours rather than minutes. That is expected for this
    hardware, not a defect of the tooling, and the reasons are the very costs Stargazer sets out to estimate.


  

    

      
Per modelItemsRetestCalls

      
        
Issues602080

        
Repositories301040

        
Two models240

      
    

  

  
Defaults of refresh_pipeline.sh (SAMPLE_N, ISSUE_RETEST, REPO_LIMIT, REPO_RETEST). One item per call, one call at a time.


  
Measured in the first run

  

  


  

    

      Memory
      
The models do not fit in GPU memory

      
The laptop GPU (RTX 4050 Laptop) has 6 GB. gpt-oss:20b is 13 GB, so more than half of it runs from
        system RAM; qwen3:8b (5.2 GB) almost fits, but the cache for its 40,960-token context spills. Decoding is
        then bound by memory bandwidth, not compute.

    

    

      Prefill
      
Long issues take minutes to read

      
Some issue bodies run to tens of thousands of tokens and are sent whole. The laptop needs minutes just to process
        such a prompt before the first output token appears.

    

    

      Reasoning
      
Both models think before answering

      
With thinking enabled, each model writes hundreds to thousands of reasoning tokens before its JSON answer, at
        laptop decode speed.

    

    

      Throughput
      
No parallelism

      
There is a single GPU, so items are rated one after another. The measured wall time of each call is kept with the
        run.

    

  

  
Model sizes as reported by ollama list; context length by ollama show.


  

    Why not paste the sheet into a chat app
    
It would answer in seconds, mostly because it runs on datacenter GPUs, but it fails three requirements: it is a
      single model; all items share one context, so one label can influence the next and nothing guarantees each item was
      read in full; and it leaves no record to regenerate from (no frozen prompt, no protocol hash, no raw answers, no
      replay). A number that cannot be regenerated cannot go into the paper.

  


  

    

      
Faster optionTrade-off

      
        
The same design through a paid API (--provider anthropic or gemini)
          Minutes instead of hours, at a monetary cost. Worth it if the pre-screen is extended to the whole filtered corpus.

        
Reasoning off, or a smaller model that fits the GPU
          Faster, but a weaker rater. That is a change of method and has to be documented.

        
A smaller context window
          Not recommended: long issues would no longer be sent, biasing the sample toward short issues.

      
    

  


  
  
Guarantees


  
Each property below is enforced in code and covered by a test. The suite runs offline, with no token and no
    gitignored corpus.


  
  
A mutation check confirms the suite has teeth: breaking the guard, the evidence allow-list, the cache key or
    the enum check each makes it fail. The Kappa code itself moved to
    common/agreement.py,
    and a golden-file test holds the human Kappa report to its output before the move.


  
  
Unattended refresh, from selection to pre-screen


  
The pre-screen is the last step of
    scripts/msr/tools/refresh_pipeline.sh,
    which rebuilds the collection from repository selection onwards without anyone at the keyboard.


  
    
    
  1. 00 · Selectselect_repos.py → bias_report.py → split_screening.py, the screening sheets for the pilot rows
    2. 
    
  2. 01 · Minemine_issues.py over the repository list; started fresh once, resumed afterwards
    3. 
    
  3. 02 · Filterfilter_issues.py, then inverse_filter.py while its audit sheet is still unlabelled
    4. 
    
  4. 05 · Funnelbuild_funnel.py, the counts on the funnel page
    5. 
    
  5. Pre-screena new stratified sample → llm_evidence.py → llm_rate.py for two local models, both funnels, with retest → llm_agreement.py
    6. 
  


  
Every step leaves a marker when it succeeds, so running the script again resumes at the first unfinished step. The miner
    also resumes at the query it stopped at instead of replaying the whole query plan, which on the 86-repository
    collection would cost most of a day of rate-limited API calls. The previous corpus is backed up before
    mining starts, and nothing is committed: the run ends with files to review.


  
tmux new-session -d -s msr-refresh \
  "systemd-inhibit --what=sleep:idle:handle-lid-switch --who=msr-refresh \
     --why='MSR pipeline refresh' --mode=block \
     bash scripts/msr/tools/refresh_pipeline.sh"

  
Detached in tmux and immune to sleep and to closing the lid. GITHUB_TOKEN comes from the environment or from gh auth token and is never printed or written to a file.


  
  
Reproduce


  

    Reproduce
    

      
Scripts
scripts/msr/03_annotation/llm_evidence.py, llm_rate.py, scripts/msr/04_agreement/llm_agreement.py

      
Shared code
common/prescreen.py (funnels, evidence, parsing, guard), common/llm_providers.py, common/agreement.py

      
Output
data/msr/llm_prescreen/<run_id>/

      
Guide
03_annotation/README.md, section LLM pre-screen; the /msr-llm-prescreen skill

      
Tests
python -m pytest scripts/msr/tests, offline

    

    
P=.venv/bin/python
# 1. freeze the evidence (a new protocol needs a new run id)
$P scripts/msr/03_annotation/llm_evidence.py --run-id R --funnel issues
GITHUB_TOKEN=... $P scripts/msr/03_annotation/llm_evidence.py --run-id R --funnel repos --limit 30

# 2. one rater per model; re-running resumes from the cache, --replay rebuilds offline
$P scripts/msr/03_annotation/llm_rate.py --run-id R --funnel issues \
    --rater gptoss --provider ollama --model gpt-oss:20b --think medium --retest 20
$P scripts/msr/03_annotation/llm_rate.py --run-id R --funnel issues \
    --rater qwen --provider ollama --model qwen3:8b --think true --retest 20

# 3. agreement, both funnels
$P scripts/msr/04_agreement/llm_agreement.py --run-id R --raters gptoss qwen

  


  
  
Limits