What two devices say about on-device inference cost

Three findings

Findings as stated in methodology.tex §Summary; the numbers are recomputed from data/.

Reproducing Stargazer v1

Stargazer v1 (Dantas Cruz et al., SMDS 2020) estimated the performance of clustering applications on an edge cluster from static code complexity and input size. Before adapting that method to language models, the team re-ran its released regression scripts on the v1 dataset: — rows of clustering benchmarks repeated on a Raspberry Pi 3 cluster.

What had to be fixed

The released scripts select columns by position. Their header comments still list the positions of an earlier, wider layout, while the released dataset has fewer columns (its header is listed in README.md). The _fixed copies in scripts/ select by the positions of the released file. They fit every target with the same polynomial degree (—), whereas v1 reports metric-specific degrees for some targets, which is the paper's explanation for the larger gaps in memory and CPU utilization.

Trained on the same data for duration only, the original DNN reached R² = —, against — for the polynomial fit (methodology.tex). For one target and two input features, the network adds little over a polynomial; its value in v1 was predicting several dependent metrics jointly.

python <repo>/scripts/ml_regression_duration_fixed.py

Testbed

Three instruction-tuned Qwen2.5 checkpoints (—), all quantized to Q4_K_M and distributed as GGUF. Holding the family and the quantization fixed and varying only parameter count follows v1's one-variable-at-a-time design. Q4_K_M is llama.cpp's default recommendation for on-device use; the — checkpoint takes — bytes at Q4_K_M against — at fp16, — smaller.

On both devices llama.cpp was built from source and ran on the CPU only for the main runs; the backend comparison in §6.5 is the one exception. Memory bandwidth was measured rather than taken from vendor specifications, because the roofline estimate in §6.4 needs the bandwidth a single thread actually sustains. On the phone, scripts/bandwidth_test.c copies an array of — doubles — times and reads back a checksum so the compiler cannot drop the copy; on the laptop an equivalent NumPy measurement was used.

clang -O2 scripts/bandwidth_test.c -o bandwidth_test
./bandwidth_test

Measurement protocol

• Speed. llama-cli with a fixed prompt, single-turn mode (-st) and a token budget (-n). Prefill and decode throughput are the rates llama-cli reports itself; no extra timing was added.
• Memory. The same command under /usr/bin/time -v; peak memory is its Maximum resident set size, which covers the whole process, from model load to the end of generation.
• Energy, laptop. The Intel RAPL package counter (/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj) read immediately before and after each run. Needs root, and gives a hardware counter rather than an estimate.
• Energy, phone. dumpsys batterystats was refused in an unrooted Termux session without ADB. The fallback is the battery percentage drop over a batch of runs, converted to joules with the rated — mAh capacity and a nominal — V cell. The method has a resolution of —% and measures the whole device, so background activity is included (Cai et al. 2026): read these values as an upper bound.
• Prompt length. The first prompt asked for an answer of a fixed number of sentences. Models honour such instructions and stop early, which would make different -n values equivalent. Responses in that round were cut mid-word, so the —-token budget did bind there; for the length sweep in §6.3 the length instruction was removed.

Source: methodology.tex §Measurement Protocol. Run counts for energy are in the table of §6.2.

Not recorded in the repository

These details are not in the repository or in the experiment report on Overleaf. They are listed so a reader knows what a re-run cannot copy exactly.

• not recordedThe run scripts: the llama-cli invocations, the RAPL reader and the battery batch.
• not recordedThe laptop's NumPy bandwidth script.
• not recordedThe exact prompt text of each round.
• not recordedThe number of repetitions behind the speed and memory table and the length sweeps.
• not recordedThe phone's CPU cores and RAM.
• not recordedThe llama.cpp build used for the main runs (the backend session records its build).

Results

6.1 · Latency and memory

Figure 1 — Throughput by model and device, —-token budget. Separate axes: decode is an order of magnitude slower than prefill.

Figure 2 — Peak resident memory over the whole run, by model and device.

On the laptop, throughput falls steadily with model size, as expected when decode is limited by memory bandwidth. On the phone, going from 1.5B to 3B cuts prefill by — although the parameter count only grows —. Memory does not explain it: the phone's 3B peak (— GiB) is below the laptop's (— GiB). The paper attributes the drop to thermal throttling under sustained load, as reported for phones by Tummalapalli et al. (2026). No temperature was recorded, so this is an inferred explanation, not a measured one.

/usr/bin/time -v llama-cli -m <model>.gguf -p "<prompt>" -st -n —

6.2 · Energy per generated token

Figure 3 — Energy per generated token. The values of the largest model are labelled: it is where the ordering of the two devices flips.

For —, the phone spends less energy per token than the laptop despite running an order of magnitude slower, which fits the lower power envelope of a mobile system-on-chip. For — the ordering inverts. The paper reads this as the same thermal effect: once the phone lowers its clock to manage temperature, it keeps drawing power while producing fewer tokens per second. Whatever the cause, the relative efficiency of two devices is not fixed; it depends on where a model size falls against each device's limits.

6.3 · Fixed and marginal cost

Three model sizes give one point per model and device, too few to test a fit: a quadratic through three points has R² = 1 by construction. Sweeping the output length over — tokens gives three points per model and one degree of freedom beyond a line, so the fit can fail. Energy on the laptop was regressed on the length n:

E(n) = E₀ + c · n

where E₀ is the fixed cost of loading the model and c the marginal energy per generated token.

Figure 4 — Measured energy (markers) and the fitted line for each model. The hollow marker at zero tokens is the fitted fixed cost E₀.

All three fits reach R² ≥ — with a genuine held-out degree of freedom. The split is more informative than the averages of §6.2, which mix a cost paid once with a cost paid per token; the two grow with model size at different rates. Over the same sweep, peak memory on the laptop moved by at most —: memory belongs to the loaded model, while latency and energy also depend on the workload.

Figure 5 — The same sweep on the phone. On a stable device decode speed would stay roughly flat as the output grows.

The phone did not give a comparably clean picture. Decode throughput was —, and peak memory moved by up to — across lengths, two orders of magnitude more than on the laptop. The paper reports this as a finding in its own right: the reliability with which cost can be measured differs between devices, which bears on how much calibration data a per-device predictor would need.

python scripts/site/build_experiments_data.py

6.4 · Roofline validation

Following Imai et al. (2024), an analytical estimate needs no training data: when decode is limited by memory bandwidth, tokens per second ≈ bandwidth ÷ model size. The bandwidths are the measured ones (— GB/s on the laptop, — GB/s on the phone).

Figure 6 — Ratio of measured to roofline-predicted decode speed. The vertical line marks a perfect prediction.

On the laptop the estimate under-predicts by a roughly constant factor (—), which the paper attributes to the bandwidth benchmark using one thread while llama.cpp decodes with several. A linear calibration, measured ≈ — + — × predicted, fits the three laptop points with R² = —, one degree of freedom left (methodology.tex prints —). On the phone the ratios are —: no single factor reconciles them. The roofline premise of a stable bandwidth holds on the laptop and not on the phone, which illustrates why a cost model calibrated on one device class does not transfer to another without recalibration (Tu et al. 2026).

6.5 · CPU and GPU backends

All runs above used the CPU. Because the backend is one of the inputs of cost, a separate llama-bench session on the laptop compared the CPU with the Vulkan backend on the integrated GPU (—): —-token prefill and —-token decode, — threads, build —, run on —.

Figure 7 — Speed-up of the Vulkan backend over the CPU backend in the same session. Right of the line, the GPU is faster.

In prefill the GPU was faster on every model (—). In decode the CPU won on the smallest model (—) and the GPU on the two larger ones (—). Which backend is cheaper depends on both the model size and the phase.

Repeating the comparison on the phone failed for three reasons, recorded in the experiment report: the Vulkan backend recognised the GPU (Adreno 618) but could not initialise because the device lacks the storageBuffer16BitAccess feature; OpenCL found no driver (ICD) available to Termux; and the backend for the Hexagon NPU needs Qualcomm's proprietary SDK, which Termux cannot install.

What each experiment supports in the paper

Source: methodology.tex §Summary and relatorio_orientador_en.tex §Additional methodological considerations. Each paper's close reading: Literature; full bibliography: References.

Limitations and threats to validity

• One phone. The paper's motivating fleet has three device categories (recent neural acceleration, older GPU or CPU acceleration, constrained memory); the single phone represents one of them.
• CPU only in the main runs. No NPU was evaluated, and the GPU only on the laptop, in a separate session.
• Coarse phone energy. Whole-device battery drop at a resolution of —% is an upper bound on inference energy, not an isolated measurement.
• No temperature recorded. Thermal throttling is the paper's explanation for the phone's behaviour; it was inferred, not measured.
• Single-thread bandwidth. The bandwidth that feeds the roofline was measured on one thread, while llama.cpp decodes with several.
• Small fits. Each fit rests on three points with one degree of freedom.
• One model family and one quantization. Qwen2.5 at Q4_K_M only.
• Missing protocol details. See what is not recorded.
• v1 values as reported. The reproduction table quotes methodology.tex; the input files and script outputs behind it are not committed.

Numbers on this page are read from docs/data/experiments.json; none is typed by hand.