Arbiter/value-graph
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

value-graph 0.1.0: JSON-free, bigint-free binary snapshot layer
CI / test (push) Successful in 20s
Details
CI / publish (push) Has been skipped
Details

Browse Source 
- src/snapshot.js: compact portable wire (VGST store / VGGP graph), binary maps for
  params/pattern/objects (no JSON), buffer values (no JS bigint), CRC-32 guarded.
- createFileStore: binary per-key files (encodeEntryBytes/decodeEntryBytes).
- ValueGraph snapshot/snapshotFile/restore/loadFile; keys base64url(params-bytes).
- 60 tests (node:test + rigor incl. 'flip ANY byte never decodes silently').
This commit is contained in:
Dvorak
2026-08-04 17:37:00 -07:00
commit 1da6bc7842
12 changed files with 5228 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+299
View File
@@ -0,0 +1,299 @@
# @arbiter/value-graph
The "second graph": a graph-structured, lazy, TTL + version-invalidated store of
derived values, built on `@push-stream-std`. The authorization graph
(`@arbiter/core`) stores structural edges with possibility; the value graph
stores **value relations** computed lazily on demand — the substrate for DSL
`measure` lookups and ARRA balances.
## Node operator model
Each value relation is an **operator node** with `parents` (its `dependsOn`):
| Operator | Kind | Behavior |
|----------|------|----------|
| `source` | leaf | external value via `fn(subject, params, ctx, cb)` (e.g. ARRA) |
| `attribute` | leaf | reads a graph node field via `resolveAttribute(nodeKey, path, params, ctx, cb)` |
| `pattern` | leaf | graph-traversal value via `resolvePattern(pattern, subject, params, ctx, cb)` |
| `compute` | interior | `fn(subject, params, ctx, cb)` over parent values (`ctx.deps`) |
| `blackbox` | interior | general function of its dependencies — `fn(subject, params, ctx, cb)` with `ctx.deps` (parent values) and `ctx.entries` (full parent results `{ value, unit, at, source, fresh }`). The seam for composing an external (e.g. Overlay) query from parent edges/values |
| `fusion:<op>` | interior | combines parent values by an OWA operator |
## Callback resolver contract (no async/await/promises)
Every resolver — `source`/`compute` `fn`, `resolveAttribute`, `resolvePattern` —
is **callback oriented**, matching the push-stream discipline of `@overlay`
adapters. A resolver MUST either:
1. call `cb(err, result)` asynchronously and return `undefined`, or
2. return the result synchronously (pure compute / memoized reads).
`result` may be a value, `{ value, unit, source }`, an interval `{ lower, upper }`,
or `null`/`undefined` (→ uncached `null`). Returning a **Promise is rejected**
with a descriptive error — this library is promise-free. This contract is what
lets the future `value-graph-overlay` bridge implement resolvers over Overlay's
callback/push-stream `FederatedQueryEngine`.
### Black-box nodes
A `blackbox` node is an arbitrary function of its resolved dependencies. Inside
the resolver, `ctx.deps[parent]` is the parent's **value**, and `ctx.entries[parent]`
is the parent's **full result** (`{ value, unit, at, source, fresh }`). Use it to
compose an external query from the parent edges/values — e.g. key an Overlay query
off a `pattern` parent's edge set:
```js
vg.define('risk', {
operator: 'blackbox',
parents: ['related_edges', 'base_balance'],
fn: (subject, params, ctx, cb) => {
const edges = ctx.deps.related_edges; // array of edges from the pattern node
const balance = ctx.entries.base_balance; // full { value, unit, at, source, fresh }
overlayQuery(buildAst(subject, edges, balance), cb); // callback-based, no promises
}
});
```
**OWA fusion operators** (ADR-000): `max`, `min`, `average`, `sum` (capped 1.0),
`sum_unbounded`, `majority`, `median`, `optimistic`, `pessimistic`, `top2`,
`top3`, `priority`, `custom` (user weights), `product`. Numeric values use
position-based weights over descending order; non-numeric values fall back to
`max`/`min`/`majority`.
## Compile → optimize → run (never fused)
- `plan(subject, relation, params)` walks `parents` into a node DAG and OPTIMIZES
it (compile and optimize are one call; the graph is deliberately **never fused** —
each operator stays a separately-cacheable node):
- **Cached-subtree pruning** — a cache-safe (fresh) node becomes a **single
trivial leaf with no parents**, pruning its dependency subtree, so `run()`
reads its cache instead of re-evaluating.
- **Ghost-parent elimination** — a fusion parent that is undeclared AND uncached
is guaranteed `null`, which fusion filters anyway; it is dropped at compile time
to avoid a wasted node evaluation. (Not applied to compute/blackbox, which
receive `null` in deps and may handle it.)
- Plan metadata: `size`, `recomputeCount` (nodes `run()` evaluates),
`prunedCount` (nodes served from cache), `ghostPruned` (dropped ghosts).
- `run(plan, cb)` evaluates the DAG in dependency order, dispatching each node to
its operator and caching the result — all callback-driven, no promises.
**Parallelism**: all independent parents are fired as soon as they are reached
(one synchronous pass starts every resolver), so N independent async resolvers
(e.g. Overlay REST calls) are in flight simultaneously — verified by test
(max in-flight === N). A node reachable via multiple paths (a diamond/shared
subtree) is evaluated **exactly once per run**: later paths join the in-flight
evaluation via waiter fan-out instead of re-firing the resolver (this fixed a
real double-firing bug for async resolvers).
**Race safety**: a run only caches its results if no `set`/`invalidate` occurred
mid-run (run-epoch guard) — a slow read can never clobber an authoritative
`set`, and a pre-mutation snapshot is never cached after an `invalidate` (both
fixed + rigor-verified).
- `invalidate(subject, relation, params)` requires the relation — calling it with
only a subject now throws instead of silently clearing the whole store
(`invalidate()` with no args still clears everything).
- `invalidate`/`set` compile the **affected dependent DAG** (the relation plus
every transitive dependent), push `{ stale: true }` triggers DOWN to active
queries (both notify the affected relation's own watchers; `set` marks the
set relation's trigger with no `via`, dependents with `via`), and bump
**per-relation version stamps** — so invalidation is
**selective**: only the affected closure recomputes; unrelated relations keep
their cached entries (verified by js-rigor).
## Pull on duplex push
`query(subject, relation, params)` returns a duplex: push `{ get: true }` up
(compile + run, serialized), receive the value down. External sources push
`invalidate`/`set` to send back-channel stale triggers.
## Backing store & larger-than-memory
The graph object itself holds only its **schema** (`relations` — the operator
definitions) and **active streams**; every cached entry lives in the swappable
backing store (`get`/`set`/`delete`/`clear`/`keys`/`deletePrefix`). That is what
makes the value graph **larger-than-memory** — data volume is decoupled from RAM,
exactly the posture of the auth graph (`@arbiter/core`'s condensed typed-array
graph + planned sharded snapshots, ADR-003).
- `createMapStore()` — in-memory default (a `Map`).
- `createFileStore(dir, { maxResident })` — disk-backed reference: one **binary**
file per entry (the same JSON-free wire encoding the snapshots use), with a
bounded LRU **hot-page cache** on top. A fresh `ValueGraph` pointed at the same
directory reads cached values straight from disk without recompute — proven by
test. **No JSON, no JS bigint** on disk — large integers are `Uint8Array`.
- Any store implementing the synchronous contract is swappable — an mmap-backed
store, a remote catalog, or (later) an Overlay-backed store.
```js
import { ValueGraph, createFileStore } from '@arbiter/value-graph';
const vg = new ValueGraph({ store: createFileStore('./vg-data', { maxResident: 256 }) });
```
### Compact snapshots, read from disk, transportable
A snapshot is a **single compact, CRC-guarded binary file** — no paths, no
machine-dependent encoding, **no JSON and no JS bigint on the wire** (large
integers are raw `Uint8Array` bytes; params/pattern/objects are deterministic
binary maps) — so a graph moves between processes and machines byte-for-byte.
Two flavours share one wire format:
- **Store snapshot** — opaque `key → entry` dump of the backing store.
`store.snapshotFile(path)` / `store.restoreFile(path)` on both store flavours;
`createFileStore(dir, { loadFrom })` constructs a store already loaded from a
snapshot.
- **Graph snapshot** — the seamless artifact: **schema metadata** (operator,
parents, weights, TTL, returnType, params — minus functions) **+ every entry**.
```js
vg.snapshotFile('./graph.snap'); // stream to disk, bounded memory
const buf = vg.snapshot(); // or a portable Buffer
const vg2 = ValueGraph.restore(buf, { resolvers: { balance: () => loadBalance() } });
const vg3 = ValueGraph.loadFile('./graph.snap', { resolvers: { balance: () => loadBalance() } });
```
`restore`/`loadFile` re-declare the schema and load entries with their version
re-stamped, so a restored cache is served **without recompute** (subject to TTL —
old snapshots expire on the wall-clock like any other entry). Functions cannot be
serialized: re-register resolvers via `{ resolvers }`. A corrupt or truncated
snapshot **throws** (magic/version/CRC/parse guards) — it never silently mis-reads.
## Typed values (DSL-declared schema)
The graph carries **schema metadata** from the evidence DSL: `define(rel, { returnType, params })`
records the declared return type (a primitive like `number`/`string`/`boolean`/`buffer`/
`timestamp`/`array`/`interval`, or an entity/Definition name treated as a reference key)
and parameter types. `relationSpec(rel)` introspects it. **There is no JS `bigint`
type** — large integers are `buffer` (`Uint8Array`), matching the JSON-free wire.
- `set(...)` validates the value against `returnType` (untyped relations are
backward-compatible and skip validation).
- Resolver results are validated against `returnType` too — a wrong-typed resolver
fails loudly instead of caching garbage.
- **`NaN` is rejected** at both the `set` and resolver-result boundaries — it is
not a meaningful derived value and would poison downstream OWA.
- `validateValueType(value, type)` is exported for consumers (the DSLValueGraph
integration uses it for binding + value checks).
This is the storage/typing substrate for `DSLValueGraph` in `@arbiter/evidence-dsl`:
measures declared in the DSL become typed value-graph nodes; attributes that are
neither cached nor computed are stored via `setValue`; `measure()`/`runtime.measure()`
retrieve through the graph.
## Example
```js
import { ValueGraph } from '@arbiter/value-graph';
const vg = new ValueGraph({
resolveAttribute: (nodeKey, path, params, ctx, cb) => cb(null, graphNodes.get(nodeKey)?.[path])
});
vg.define('base_balance', { operator: 'source', fn: (s, p, ctx, cb) => cb(null, ledgerBalance(s)) });
vg.define('risk_adjusted', { operator: 'compute', parents: ['base_balance'], fn: (s, p, { deps }) => deps.base_balance * 0.9 });
vg.define('effective_clearance', { operator: 'fusion:max', parents: ['base', 'role', 'group'] });
vg.define('avg_score', { operator: 'fusion:custom', parents: ['rep', 'activity', 'verif'], weights: [0.6, 0.3, 0.1] });
const plan = vg.plan('tenant:acme', 'risk_adjusted', {});
vg.run(plan, (err, value) => { /* { value, unit, at, source, fresh } */ });
```
## Deep testing (js-rigor)
`test/value-graph.rigor.test.js` (`npm run test:rigor`) is a js-rigor suite:
- **OWA properties** — `owa`/`weightFor` invariants (sum/avg/max/min/median/top2/
product/optimistic/pessimistic/custom, string fallbacks) + a `sum` oracle. This
caught and fixed two real edge cases: `weightFor('top2'|'top3', n)` over-produced
weight vectors for `n` smaller than the named positions, and `weightFor(op, 0)`
returned `[1]` for `max`/`min` instead of `[]`.
- **OWA oracle conformance** — an *independent* reimplementation of the ADR-000 OWA
formula (`refOwaExact`, mirrored arithmetic for bit-exact `===`) confirms every
operator: exact oracles for sum/max/min/product/median/string fallbacks + capped
sum, and **arbitrary operator combinations** (a frequency-weighted pick over all
14 operators × random arrays × random weights/priorities) must match the reference.
- **Fusion DAG oracles** — the graph's fusion path must equal raw `owa` for any
operator, and **composed DAGs** (fusion-of-fusion, compute→fusion chains) must
equal the bottom-up reference over the same operators.
- **More correctness oracles** — mixed fusion+compute topologies vs the bottom-up
reference; **invalidation correctness** (changing a source and invalidating it
yields the correct new root value across the dependent subtree); **cache-key
isolation** (subjects never share entries, each computes once); **stale-on-error
fallback** (a failing resolver after TTL expiry returns the cached value with
`fresh:false` instead of erroring); **value-type round-trips** (array/interval/
buffer survive the disk-backed store into a fresh graph instance); **blackbox
over fusion** (`fusion(a,b)·k` matches the reference); and **selective
invalidation** (invalidating/setting an unrelated relation leaves its cached
entry untouched; a sibling independent subtree stays cached).
- **Universal invariants** — for *every* operator: empty→`0`, singleton→element,
sum/sum_unbounded/product exact, weighted operators bounded within `[min,max]`,
and monotone non-decreasing under a `+1` bump. Plus: **plan exactness**
(fully-cached plan is one leaf; invalidation expands to exactly the affected
closure with unrelated leaves still trivial); **async serialization** (two
concurrent `{get}` pushes with an async resolver deliver both values and the
second is served from cache — resolver called once); **params cache-key
isolation**; **cross-subject invalidation correctness**; **invalidateAll clears
every subject**; **ghost parents are filtered from fusions**; **attribute/
pattern hooks receive the documented arguments**; **TTL boundary is exact**
(fresh inside, stale at the window, `ttl:0` never expires); **`set` preserves
unit/source**; in-graph string fusion fallbacks + `capSum`; and **duplicate
parents are a single dependency** (parents are a set).
- **Random-topology DAG oracles** — three mid fusion nodes over randomly-picked
source pairs plus a root fusion, all with random operators, must match the
independent bottom-up reference (`refOwaExact`), including the duplicate-parent
set semantics.
- **Complexity verification** — `run()` on a deep compute chain is `O(n)` resolver
calls (no recomputation blowup), `plan()` compiles a deep chain in `O(n)`, and
`weightFor` allocates `O(n)` weights (deterministic cost metrics).
- **Compile-time optimization** — ghost parents are eliminated at compile time
(semantics preserved for every operator vs `refOwaExact`), a fully-cached plan is
a single trivial leaf with zero recomputes, and an invalidated plan recomputes
exactly the affected closure while serving the untouched sibling from cache.
- **Graph caching model (formal conformance)** — `rigor.model.check` runs random
`get`/`set`/`invalidate`/`mutate` sequences against a reference model of the
per-relation-version caching contract and the real graph; every value must match.
Pins the exact semantics: `set` writes fresh without bumping the relation version,
`invalidate` bumps it and recomputes, and a backing mutation without invalidation
stays cached until the next invalidation/TTL.
- **Query duplex lifecycle** — a randomized `get`/`invalidate` stream delivers only
correct values and well-formed stale triggers; `end()` cleans up so post-end
writes are silent no-ops (including `invalidate` pushes to the closed query); and
`abort()` surfaces its error on the stream.
- **Store-fault robustness** — a throwing store must never hang or crash the graph:
a store whose `set` always fails still serves correct values (best-effort uncached
serve — this fixed a real hang where a run-time `store.set` failure was swallowed
by the resolver guard), a flaky store recovers (later gets are cached), and a
store whose `get` fails surfaces the error via callback. The universal OWA
invariants also pin the constant-array case (all-equal inputs → the value for
weighted operators).
- **Algebraic OWA invariants** — results are **invariant under input order** (any
operator: sort-based), and **max/min/average/median are invariant under multiset
duplication** (each element duplicated; FP-tolerance for average).
- **Params-scoped invalidation & CSE** — invalidating one params set deletes that
key while every subject/params value stays correct; a **shared subtree** in a
diamond DAG is computed once across two roots and once across the post-
invalidation recompute (no redundant evaluation); and `invalidate` clears an
eager `set` value so the next get recomputes from the resolver.
- **Action-level fault injection** — under injected `rigor.faults` (throw on the
get action) the callback contract still holds: correct value or surfaced error,
never a hang.
- **Parallel firing & async CSE** — N independent async resolvers fire
simultaneously (max in-flight === N), each exactly once; a diamond DAG with
async resolvers fires its shared node exactly once (no duplicate REST calls);
and waiter fan-out joins at every level of a deep shared subtree.
- **Stateful protocol, concurrency, edges, benchmark** — a `rigor.object` protocol
campaign (with `before`/`after`/`between`) pins "after `set(v)` every `get`
returns `v` until `invalidate`"; **concurrent direct gets** on one graph each
complete independently and correctly (N calls for N gets, no torn state); a run
error inside a query duplex is pushed down as `{ error }`; `set` rejects
unsupported value shapes; resolvers propagate `{ value, unit, source }` onto the
delivered entry; and a loose **latency benchmark** smoke guards the fast path.
- **Graph semantics** — caching (second get computes nothing), invalidation cascade,
`set` override, plan pruning, TTL expiry, `blackbox` deps/entries.
- **Callback contract** — handler-based: sync + async resolver delivery, resolver
error propagation, Promise-return rejection, duplex pull (two gets), and stale
trigger pushed between values on `invalidate`.
- **Store contract** — model-based conformance (`rigor.model.check`) of
`createMapStore` and `createFileStore` against a reference `Map` under random
`set`/`get`/`delete`/`clear`/`deletePrefix` sequences.
## License
ISC