The CI benchmark gate was flagging wild run-to-run 'regressions'/'improvements'
(-23%..-83% on unchanged code) because sub-ms checks were measured from ~5
samples with a global.gc() injected between every iteration:
- uncertaintyThreshold 0.99 -> 0.1: the loop now keeps sampling until variance
actually tightens instead of 'passing' at the first check.
- minSamples 0 -> 200, maxSamples 200 -> 2000: real sample floor + headroom.
- gcBetweenSamples true -> false: per-sample GC dominated sub-ms timings.
- warmup phase: each hot path runs to steady state (JIT, lazy index, caches)
before sampling, eliminating the bimodal ~4us vs ~20us distribution.
- BATCH=100 for sub-ms checks: jitter amortizes across a batch per sample; the
relative comparison stays exact because the baseline uses the same batch.
- minimum-change threshold (MIN_HIGH_REGRESSION_PERCENT, default 10): a
high-severity flag only fails CI when the change exceeds run-to-run noise.
Result: within-run p95 spread is now ~5% instead of ~100x. Residual cross-run
variance on loaded shared runners (this machine: load ~19) is environmental —
the baseline alphaCuts capture per-run spread, not machine-load swings.
README: purpose-first (possibility not boolean, caller owns evidence/time),
install, verified quick start, concepts (result shape, overlays, temporal
context), API table, development commands, design notes.
benchmark: scripts/benchmark.js on @tenere/benchmark-lib — eight contours
(direct/union/denied/meta/overlay/binary checks, snapshot build/restore),
committed .rigor-baseline.json, exit 1 on high-severity regressions.
CI: benchmark job compares on push (continue-on-error), re-saves baseline
and uploads it as an artifact on tags; publish now depends on benchmark
passing as well as test.
John Dvorak