Dvorak
1da6bc7842
- src/snapshot.js: compact portable wire (VGST store / VGGP graph), binary maps for params/pattern/objects (no JSON), buffer values (no JS bigint), CRC-32 guarded. - createFileStore: binary per-key files (encodeEntryBytes/decodeEntryBytes). - ValueGraph snapshot/snapshotFile/restore/loadFile; keys base64url(params-bytes). - 60 tests (node:test + rigor incl. 'flip ANY byte never decodes silently').
300 lines
18 KiB
Markdown
300 lines
18 KiB
Markdown
# @arbiter/value-graph
|
||
|
||
The "second graph": a graph-structured, lazy, TTL + version-invalidated store of
|
||
derived values, built on `@push-stream-std`. The authorization graph
|
||
(`@arbiter/core`) stores structural edges with possibility; the value graph
|
||
stores **value relations** computed lazily on demand — the substrate for DSL
|
||
`measure` lookups and ARRA balances.
|
||
|
||
## Node operator model
|
||
|
||
Each value relation is an **operator node** with `parents` (its `dependsOn`):
|
||
|
||
| Operator | Kind | Behavior |
|
||
|----------|------|----------|
|
||
| `source` | leaf | external value via `fn(subject, params, ctx, cb)` (e.g. ARRA) |
|
||
| `attribute` | leaf | reads a graph node field via `resolveAttribute(nodeKey, path, params, ctx, cb)` |
|
||
| `pattern` | leaf | graph-traversal value via `resolvePattern(pattern, subject, params, ctx, cb)` |
|
||
| `compute` | interior | `fn(subject, params, ctx, cb)` over parent values (`ctx.deps`) |
|
||
| `blackbox` | interior | general function of its dependencies — `fn(subject, params, ctx, cb)` with `ctx.deps` (parent values) and `ctx.entries` (full parent results `{ value, unit, at, source, fresh }`). The seam for composing an external (e.g. Overlay) query from parent edges/values |
|
||
| `fusion:<op>` | interior | combines parent values by an OWA operator |
|
||
|
||
## Callback resolver contract (no async/await/promises)
|
||
|
||
Every resolver — `source`/`compute` `fn`, `resolveAttribute`, `resolvePattern` —
|
||
is **callback oriented**, matching the push-stream discipline of `@overlay`
|
||
adapters. A resolver MUST either:
|
||
|
||
1. call `cb(err, result)` asynchronously and return `undefined`, or
|
||
2. return the result synchronously (pure compute / memoized reads).
|
||
|
||
`result` may be a value, `{ value, unit, source }`, an interval `{ lower, upper }`,
|
||
or `null`/`undefined` (→ uncached `null`). Returning a **Promise is rejected**
|
||
with a descriptive error — this library is promise-free. This contract is what
|
||
lets the future `value-graph-overlay` bridge implement resolvers over Overlay's
|
||
callback/push-stream `FederatedQueryEngine`.
|
||
|
||
### Black-box nodes
|
||
|
||
A `blackbox` node is an arbitrary function of its resolved dependencies. Inside
|
||
the resolver, `ctx.deps[parent]` is the parent's **value**, and `ctx.entries[parent]`
|
||
is the parent's **full result** (`{ value, unit, at, source, fresh }`). Use it to
|
||
compose an external query from the parent edges/values — e.g. key an Overlay query
|
||
off a `pattern` parent's edge set:
|
||
|
||
```js
|
||
vg.define('risk', {
|
||
operator: 'blackbox',
|
||
parents: ['related_edges', 'base_balance'],
|
||
fn: (subject, params, ctx, cb) => {
|
||
const edges = ctx.deps.related_edges; // array of edges from the pattern node
|
||
const balance = ctx.entries.base_balance; // full { value, unit, at, source, fresh }
|
||
overlayQuery(buildAst(subject, edges, balance), cb); // callback-based, no promises
|
||
}
|
||
});
|
||
```
|
||
|
||
**OWA fusion operators** (ADR-000): `max`, `min`, `average`, `sum` (capped 1.0),
|
||
`sum_unbounded`, `majority`, `median`, `optimistic`, `pessimistic`, `top2`,
|
||
`top3`, `priority`, `custom` (user weights), `product`. Numeric values use
|
||
position-based weights over descending order; non-numeric values fall back to
|
||
`max`/`min`/`majority`.
|
||
|
||
## Compile → optimize → run (never fused)
|
||
|
||
- `plan(subject, relation, params)` walks `parents` into a node DAG and OPTIMIZES
|
||
it (compile and optimize are one call; the graph is deliberately **never fused** —
|
||
each operator stays a separately-cacheable node):
|
||
- **Cached-subtree pruning** — a cache-safe (fresh) node becomes a **single
|
||
trivial leaf with no parents**, pruning its dependency subtree, so `run()`
|
||
reads its cache instead of re-evaluating.
|
||
- **Ghost-parent elimination** — a fusion parent that is undeclared AND uncached
|
||
is guaranteed `null`, which fusion filters anyway; it is dropped at compile time
|
||
to avoid a wasted node evaluation. (Not applied to compute/blackbox, which
|
||
receive `null` in deps and may handle it.)
|
||
- Plan metadata: `size`, `recomputeCount` (nodes `run()` evaluates),
|
||
`prunedCount` (nodes served from cache), `ghostPruned` (dropped ghosts).
|
||
- `run(plan, cb)` evaluates the DAG in dependency order, dispatching each node to
|
||
its operator and caching the result — all callback-driven, no promises.
|
||
**Parallelism**: all independent parents are fired as soon as they are reached
|
||
(one synchronous pass starts every resolver), so N independent async resolvers
|
||
(e.g. Overlay REST calls) are in flight simultaneously — verified by test
|
||
(max in-flight === N). A node reachable via multiple paths (a diamond/shared
|
||
subtree) is evaluated **exactly once per run**: later paths join the in-flight
|
||
evaluation via waiter fan-out instead of re-firing the resolver (this fixed a
|
||
real double-firing bug for async resolvers).
|
||
**Race safety**: a run only caches its results if no `set`/`invalidate` occurred
|
||
mid-run (run-epoch guard) — a slow read can never clobber an authoritative
|
||
`set`, and a pre-mutation snapshot is never cached after an `invalidate` (both
|
||
fixed + rigor-verified).
|
||
- `invalidate(subject, relation, params)` requires the relation — calling it with
|
||
only a subject now throws instead of silently clearing the whole store
|
||
(`invalidate()` with no args still clears everything).
|
||
- `invalidate`/`set` compile the **affected dependent DAG** (the relation plus
|
||
every transitive dependent), push `{ stale: true }` triggers DOWN to active
|
||
queries (both notify the affected relation's own watchers; `set` marks the
|
||
set relation's trigger with no `via`, dependents with `via`), and bump
|
||
**per-relation version stamps** — so invalidation is
|
||
**selective**: only the affected closure recomputes; unrelated relations keep
|
||
their cached entries (verified by js-rigor).
|
||
|
||
## Pull on duplex push
|
||
|
||
`query(subject, relation, params)` returns a duplex: push `{ get: true }` up
|
||
(compile + run, serialized), receive the value down. External sources push
|
||
`invalidate`/`set` to send back-channel stale triggers.
|
||
|
||
## Backing store & larger-than-memory
|
||
|
||
The graph object itself holds only its **schema** (`relations` — the operator
|
||
definitions) and **active streams**; every cached entry lives in the swappable
|
||
backing store (`get`/`set`/`delete`/`clear`/`keys`/`deletePrefix`). That is what
|
||
makes the value graph **larger-than-memory** — data volume is decoupled from RAM,
|
||
exactly the posture of the auth graph (`@arbiter/core`'s condensed typed-array
|
||
graph + planned sharded snapshots, ADR-003).
|
||
|
||
- `createMapStore()` — in-memory default (a `Map`).
|
||
- `createFileStore(dir, { maxResident })` — disk-backed reference: one **binary**
|
||
file per entry (the same JSON-free wire encoding the snapshots use), with a
|
||
bounded LRU **hot-page cache** on top. A fresh `ValueGraph` pointed at the same
|
||
directory reads cached values straight from disk without recompute — proven by
|
||
test. **No JSON, no JS bigint** on disk — large integers are `Uint8Array`.
|
||
- Any store implementing the synchronous contract is swappable — an mmap-backed
|
||
store, a remote catalog, or (later) an Overlay-backed store.
|
||
|
||
```js
|
||
import { ValueGraph, createFileStore } from '@arbiter/value-graph';
|
||
|
||
const vg = new ValueGraph({ store: createFileStore('./vg-data', { maxResident: 256 }) });
|
||
```
|
||
|
||
### Compact snapshots, read from disk, transportable
|
||
|
||
A snapshot is a **single compact, CRC-guarded binary file** — no paths, no
|
||
machine-dependent encoding, **no JSON and no JS bigint on the wire** (large
|
||
integers are raw `Uint8Array` bytes; params/pattern/objects are deterministic
|
||
binary maps) — so a graph moves between processes and machines byte-for-byte.
|
||
Two flavours share one wire format:
|
||
|
||
- **Store snapshot** — opaque `key → entry` dump of the backing store.
|
||
`store.snapshotFile(path)` / `store.restoreFile(path)` on both store flavours;
|
||
`createFileStore(dir, { loadFrom })` constructs a store already loaded from a
|
||
snapshot.
|
||
- **Graph snapshot** — the seamless artifact: **schema metadata** (operator,
|
||
parents, weights, TTL, returnType, params — minus functions) **+ every entry**.
|
||
|
||
```js
|
||
vg.snapshotFile('./graph.snap'); // stream to disk, bounded memory
|
||
const buf = vg.snapshot(); // or a portable Buffer
|
||
const vg2 = ValueGraph.restore(buf, { resolvers: { balance: () => loadBalance() } });
|
||
const vg3 = ValueGraph.loadFile('./graph.snap', { resolvers: { balance: () => loadBalance() } });
|
||
```
|
||
|
||
`restore`/`loadFile` re-declare the schema and load entries with their version
|
||
re-stamped, so a restored cache is served **without recompute** (subject to TTL —
|
||
old snapshots expire on the wall-clock like any other entry). Functions cannot be
|
||
serialized: re-register resolvers via `{ resolvers }`. A corrupt or truncated
|
||
snapshot **throws** (magic/version/CRC/parse guards) — it never silently mis-reads.
|
||
|
||
## Typed values (DSL-declared schema)
|
||
|
||
The graph carries **schema metadata** from the evidence DSL: `define(rel, { returnType, params })`
|
||
records the declared return type (a primitive like `number`/`string`/`boolean`/`buffer`/
|
||
`timestamp`/`array`/`interval`, or an entity/Definition name treated as a reference key)
|
||
and parameter types. `relationSpec(rel)` introspects it. **There is no JS `bigint`
|
||
type** — large integers are `buffer` (`Uint8Array`), matching the JSON-free wire.
|
||
|
||
- `set(...)` validates the value against `returnType` (untyped relations are
|
||
backward-compatible and skip validation).
|
||
- Resolver results are validated against `returnType` too — a wrong-typed resolver
|
||
fails loudly instead of caching garbage.
|
||
- **`NaN` is rejected** at both the `set` and resolver-result boundaries — it is
|
||
not a meaningful derived value and would poison downstream OWA.
|
||
- `validateValueType(value, type)` is exported for consumers (the DSLValueGraph
|
||
integration uses it for binding + value checks).
|
||
|
||
This is the storage/typing substrate for `DSLValueGraph` in `@arbiter/evidence-dsl`:
|
||
measures declared in the DSL become typed value-graph nodes; attributes that are
|
||
neither cached nor computed are stored via `setValue`; `measure()`/`runtime.measure()`
|
||
retrieve through the graph.
|
||
|
||
## Example
|
||
|
||
```js
|
||
import { ValueGraph } from '@arbiter/value-graph';
|
||
|
||
const vg = new ValueGraph({
|
||
resolveAttribute: (nodeKey, path, params, ctx, cb) => cb(null, graphNodes.get(nodeKey)?.[path])
|
||
});
|
||
|
||
vg.define('base_balance', { operator: 'source', fn: (s, p, ctx, cb) => cb(null, ledgerBalance(s)) });
|
||
vg.define('risk_adjusted', { operator: 'compute', parents: ['base_balance'], fn: (s, p, { deps }) => deps.base_balance * 0.9 });
|
||
vg.define('effective_clearance', { operator: 'fusion:max', parents: ['base', 'role', 'group'] });
|
||
vg.define('avg_score', { operator: 'fusion:custom', parents: ['rep', 'activity', 'verif'], weights: [0.6, 0.3, 0.1] });
|
||
|
||
const plan = vg.plan('tenant:acme', 'risk_adjusted', {});
|
||
vg.run(plan, (err, value) => { /* { value, unit, at, source, fresh } */ });
|
||
```
|
||
|
||
## Deep testing (js-rigor)
|
||
|
||
`test/value-graph.rigor.test.js` (`npm run test:rigor`) is a js-rigor suite:
|
||
- **OWA properties** — `owa`/`weightFor` invariants (sum/avg/max/min/median/top2/
|
||
product/optimistic/pessimistic/custom, string fallbacks) + a `sum` oracle. This
|
||
caught and fixed two real edge cases: `weightFor('top2'|'top3', n)` over-produced
|
||
weight vectors for `n` smaller than the named positions, and `weightFor(op, 0)`
|
||
returned `[1]` for `max`/`min` instead of `[]`.
|
||
- **OWA oracle conformance** — an *independent* reimplementation of the ADR-000 OWA
|
||
formula (`refOwaExact`, mirrored arithmetic for bit-exact `===`) confirms every
|
||
operator: exact oracles for sum/max/min/product/median/string fallbacks + capped
|
||
sum, and **arbitrary operator combinations** (a frequency-weighted pick over all
|
||
14 operators × random arrays × random weights/priorities) must match the reference.
|
||
- **Fusion DAG oracles** — the graph's fusion path must equal raw `owa` for any
|
||
operator, and **composed DAGs** (fusion-of-fusion, compute→fusion chains) must
|
||
equal the bottom-up reference over the same operators.
|
||
- **More correctness oracles** — mixed fusion+compute topologies vs the bottom-up
|
||
reference; **invalidation correctness** (changing a source and invalidating it
|
||
yields the correct new root value across the dependent subtree); **cache-key
|
||
isolation** (subjects never share entries, each computes once); **stale-on-error
|
||
fallback** (a failing resolver after TTL expiry returns the cached value with
|
||
`fresh:false` instead of erroring); **value-type round-trips** (array/interval/
|
||
buffer survive the disk-backed store into a fresh graph instance); **blackbox
|
||
over fusion** (`fusion(a,b)·k` matches the reference); and **selective
|
||
invalidation** (invalidating/setting an unrelated relation leaves its cached
|
||
entry untouched; a sibling independent subtree stays cached).
|
||
- **Universal invariants** — for *every* operator: empty→`0`, singleton→element,
|
||
sum/sum_unbounded/product exact, weighted operators bounded within `[min,max]`,
|
||
and monotone non-decreasing under a `+1` bump. Plus: **plan exactness**
|
||
(fully-cached plan is one leaf; invalidation expands to exactly the affected
|
||
closure with unrelated leaves still trivial); **async serialization** (two
|
||
concurrent `{get}` pushes with an async resolver deliver both values and the
|
||
second is served from cache — resolver called once); **params cache-key
|
||
isolation**; **cross-subject invalidation correctness**; **invalidateAll clears
|
||
every subject**; **ghost parents are filtered from fusions**; **attribute/
|
||
pattern hooks receive the documented arguments**; **TTL boundary is exact**
|
||
(fresh inside, stale at the window, `ttl:0` never expires); **`set` preserves
|
||
unit/source**; in-graph string fusion fallbacks + `capSum`; and **duplicate
|
||
parents are a single dependency** (parents are a set).
|
||
- **Random-topology DAG oracles** — three mid fusion nodes over randomly-picked
|
||
source pairs plus a root fusion, all with random operators, must match the
|
||
independent bottom-up reference (`refOwaExact`), including the duplicate-parent
|
||
set semantics.
|
||
- **Complexity verification** — `run()` on a deep compute chain is `O(n)` resolver
|
||
calls (no recomputation blowup), `plan()` compiles a deep chain in `O(n)`, and
|
||
`weightFor` allocates `O(n)` weights (deterministic cost metrics).
|
||
- **Compile-time optimization** — ghost parents are eliminated at compile time
|
||
(semantics preserved for every operator vs `refOwaExact`), a fully-cached plan is
|
||
a single trivial leaf with zero recomputes, and an invalidated plan recomputes
|
||
exactly the affected closure while serving the untouched sibling from cache.
|
||
- **Graph caching model (formal conformance)** — `rigor.model.check` runs random
|
||
`get`/`set`/`invalidate`/`mutate` sequences against a reference model of the
|
||
per-relation-version caching contract and the real graph; every value must match.
|
||
Pins the exact semantics: `set` writes fresh without bumping the relation version,
|
||
`invalidate` bumps it and recomputes, and a backing mutation without invalidation
|
||
stays cached until the next invalidation/TTL.
|
||
- **Query duplex lifecycle** — a randomized `get`/`invalidate` stream delivers only
|
||
correct values and well-formed stale triggers; `end()` cleans up so post-end
|
||
writes are silent no-ops (including `invalidate` pushes to the closed query); and
|
||
`abort()` surfaces its error on the stream.
|
||
- **Store-fault robustness** — a throwing store must never hang or crash the graph:
|
||
a store whose `set` always fails still serves correct values (best-effort uncached
|
||
serve — this fixed a real hang where a run-time `store.set` failure was swallowed
|
||
by the resolver guard), a flaky store recovers (later gets are cached), and a
|
||
store whose `get` fails surfaces the error via callback. The universal OWA
|
||
invariants also pin the constant-array case (all-equal inputs → the value for
|
||
weighted operators).
|
||
- **Algebraic OWA invariants** — results are **invariant under input order** (any
|
||
operator: sort-based), and **max/min/average/median are invariant under multiset
|
||
duplication** (each element duplicated; FP-tolerance for average).
|
||
- **Params-scoped invalidation & CSE** — invalidating one params set deletes that
|
||
key while every subject/params value stays correct; a **shared subtree** in a
|
||
diamond DAG is computed once across two roots and once across the post-
|
||
invalidation recompute (no redundant evaluation); and `invalidate` clears an
|
||
eager `set` value so the next get recomputes from the resolver.
|
||
- **Action-level fault injection** — under injected `rigor.faults` (throw on the
|
||
get action) the callback contract still holds: correct value or surfaced error,
|
||
never a hang.
|
||
- **Parallel firing & async CSE** — N independent async resolvers fire
|
||
simultaneously (max in-flight === N), each exactly once; a diamond DAG with
|
||
async resolvers fires its shared node exactly once (no duplicate REST calls);
|
||
and waiter fan-out joins at every level of a deep shared subtree.
|
||
- **Stateful protocol, concurrency, edges, benchmark** — a `rigor.object` protocol
|
||
campaign (with `before`/`after`/`between`) pins "after `set(v)` every `get`
|
||
returns `v` until `invalidate`"; **concurrent direct gets** on one graph each
|
||
complete independently and correctly (N calls for N gets, no torn state); a run
|
||
error inside a query duplex is pushed down as `{ error }`; `set` rejects
|
||
unsupported value shapes; resolvers propagate `{ value, unit, source }` onto the
|
||
delivered entry; and a loose **latency benchmark** smoke guards the fast path.
|
||
- **Graph semantics** — caching (second get computes nothing), invalidation cascade,
|
||
`set` override, plan pruning, TTL expiry, `blackbox` deps/entries.
|
||
- **Callback contract** — handler-based: sync + async resolver delivery, resolver
|
||
error propagation, Promise-return rejection, duplex pull (two gets), and stale
|
||
trigger pushed between values on `invalidate`.
|
||
- **Store contract** — model-based conformance (`rigor.model.check`) of
|
||
`createMapStore` and `createFileStore` against a reference `Map` under random
|
||
`set`/`get`/`delete`/`clear`/`deletePrefix` sequences.
|
||
|
||
## License
|
||
|
||
ISC
|