Case studies / webpack/webpack
The point of this exhibit is the gap between those two numbers. Our autoresearch
engine estimated a +45.6% wall-clock win for a composed candidate against
webpack/webpack. Ten interleaved live runs on the substrate a real client actually
uses — GitHub-hosted CI — verified a statistically significant but far smaller win:
+14.1%. The change is real. The estimate was roughly three times too optimistic.
Only the measurement is safe to guarantee — and only measurement on the real substrate produced
it.
The verified result
| Candidate | setup-node-cache + remove-needs, composed |
| Estimate (autoresearch) | +45.6% wall-clock |
| Verified p50 wall-clock delta | +14.1% (baseline p50 2757s → candidate p50 2369s) |
| Mann-Whitney U p-value | 0.019 (< 0.05 → gate kept / verified) |
| p50 bootstrap CI | [≈0%, +34.1%] |
| p95 wall-clock delta | +21.0% (no tail regression) |
| N | 10 baseline / 10 candidate, interleaved — not provisional |
calibration_ratio (verified ÷ estimated) | 0.31 |
Read it honestly, both ways. It passed: the candidate genuinely makes webpack's CI faster, and the Mann-Whitney rank test is significant because the candidate distribution sits consistently below baseline. And it's a large calibration miss: +14.1% is under a third of the +45.6% estimate, and the p50 bootstrap CI's lower bound sits right at zero — so while the effect is significant by the rank test, its magnitude carries real uncertainty. A guarantee written off the +45.6% estimate would have been off by roughly 3×; a guarantee written off the +14.1% measured number is safe.
The critical-path model estimated the saving from mean/median per-task durations on the dependency graph. Real GitHub-hosted wall-clock is set by matrix-tail and queue/runner-start latency variance that don't shrink proportionally when you cache — the kind of substrate noise a static model can't see and a live measurement catches automatically.
This is why the diagnostic step exists, and why it runs on your CI specifically, not a generic model. An estimate tells you where to look. A measurement tells you what to guarantee.