The CI benchmark gate was flagging wild run-to-run 'regressions'/'improvements'
(-23%..-83% on unchanged code) because sub-ms checks were measured from ~5
samples with a global.gc() injected between every iteration:
- uncertaintyThreshold 0.99 -> 0.1: the loop now keeps sampling until variance
actually tightens instead of 'passing' at the first check.
- minSamples 0 -> 200, maxSamples 200 -> 2000: real sample floor + headroom.
- gcBetweenSamples true -> false: per-sample GC dominated sub-ms timings.
- warmup phase: each hot path runs to steady state (JIT, lazy index, caches)
before sampling, eliminating the bimodal ~4us vs ~20us distribution.
- BATCH=100 for sub-ms checks: jitter amortizes across a batch per sample; the
relative comparison stays exact because the baseline uses the same batch.
- minimum-change threshold (MIN_HIGH_REGRESSION_PERCENT, default 10): a
high-severity flag only fails CI when the change exceeds run-to-run noise.
Result: within-run p95 spread is now ~5% instead of ~100x. Residual cross-run
variance on loaded shared runners (this machine: load ~19) is environmental —
the baseline alphaCuts capture per-run spread, not machine-load swings.
The prod-gating measurement (complex-query-bench: tuple-to-userset,
chains, defeasible, comparators, OWA, nested fusion, mixed unions over
cold traffic with binary parity) was committed but never ran in CI. The
benchmark job now runs it alongside the possibilistic baseline compare,
uploads the results artifact, and surfaces both in the run summary.
Also: refreshed the committed .rigor-baseline.json (engine unchanged
since the tag; the runner's canonical baseline lives in its artifact
store), and added npm run benchmark:complex-query for local runs.
README: purpose-first (possibility not boolean, caller owns evidence/time),
install, verified quick start, concepts (result shape, overlays, temporal
context), API table, development commands, design notes.
benchmark: scripts/benchmark.js on @tenere/benchmark-lib — eight contours
(direct/union/denied/meta/overlay/binary checks, snapshot build/restore),
committed .rigor-baseline.json, exit 1 on high-severity regressions.
CI: benchmark job compares on push (continue-on-error), re-saves baseline
and uploads it as an artifact on tags; publish now depends on benchmark
passing as well as test.
John Dvorak