The CI benchmark gate was flagging wild run-to-run 'regressions'/'improvements'
(-23%..-83% on unchanged code) because sub-ms checks were measured from ~5
samples with a global.gc() injected between every iteration:
- uncertaintyThreshold 0.99 -> 0.1: the loop now keeps sampling until variance
actually tightens instead of 'passing' at the first check.
- minSamples 0 -> 200, maxSamples 200 -> 2000: real sample floor + headroom.
- gcBetweenSamples true -> false: per-sample GC dominated sub-ms timings.
- warmup phase: each hot path runs to steady state (JIT, lazy index, caches)
before sampling, eliminating the bimodal ~4us vs ~20us distribution.
- BATCH=100 for sub-ms checks: jitter amortizes across a batch per sample; the
relative comparison stays exact because the baseline uses the same batch.
- minimum-change threshold (MIN_HIGH_REGRESSION_PERCENT, default 10): a
high-severity flag only fails CI when the change exceeds run-to-run noise.
Result: within-run p95 spread is now ~5% instead of ~100x. Residual cross-run
variance on loaded shared runners (this machine: load ~19) is environmental —
the baseline alphaCuts capture per-run spread, not machine-load swings.
The Evidence DSL (ADR-000) compiles a natural DSL into core rule
configurations — it is a separate concern from the engine. The AST had
zero runtime coupling to the core (DSLCompiler takes the arbiter as a
duck-typed argument; ip-utils were the only shared code, now local to
the DSL package). This extraction removes the DSL surface from the core
artifact entirely:
- src/ast/ (748K, ~60 files) moved to @arbiter/evidence-dsl@1.0.0
- ip-utils moved with it (only the DSL consumed them)
- generate-parser script + peggy devDep moved to the DSL package
- the 8 DSL-consuming tests now import from @arbiter/evidence-dsl
(deep-path exports: DSLCompiler, parser/*, generator/*, validation/*,
interpreter/*)
- package.json gains the devDependency, drops build:ast/generate:parser
Tarball: AST-free. Rigor 251/251, full suite 838/776/0.
README: purpose-first (possibility not boolean, caller owns evidence/time),
install, verified quick start, concepts (result shape, overlays, temporal
context), API table, development commands, design notes.
benchmark: scripts/benchmark.js on @tenere/benchmark-lib — eight contours
(direct/union/denied/meta/overlay/binary checks, snapshot build/restore),
committed .rigor-baseline.json, exit 1 on high-severity regressions.
CI: benchmark job compares on push (continue-on-error), re-saves baseline
and uploads it as an artifact on tags; publish now depends on benchmark
passing as well as test.
John Dvorak