# @arbiter/value-graph

The "second graph": a graph-structured, lazy, TTL + version-invalidated store of
derived values, built on `@push-stream-std`. The authorization graph
(`@arbiter/core`) stores structural edges with possibility; the value graph
stores **value relations** computed lazily on demand — the substrate for DSL
`measure` lookups and ARRA balances.

## Node operator model

Each value relation is an **operator node** with `parents` (its `dependsOn`):

| Operator | Kind | Behavior |
|----------|------|----------|
| `source` | leaf | external value via `fn(subject, params, ctx, cb)` (e.g. ARRA) |
| `attribute` | leaf | reads a graph node field via `resolveAttribute(nodeKey, path, params, ctx, cb)` |
| `pattern` | leaf | graph-traversal value via `resolvePattern(pattern, subject, params, ctx, cb)` |
| `compute` | interior | `fn(subject, params, ctx, cb)` over parent values (`ctx.deps`) |
| `blackbox` | interior | general function of its dependencies — `fn(subject, params, ctx, cb)` with `ctx.deps` (parent values) and `ctx.entries` (full parent results `{ value, unit, at, source, fresh }`). The seam for composing an external (e.g. Overlay) query from parent edges/values |
| `fusion:<op>` | interior | combines parent values by an OWA operator |

## Callback resolver contract (no async/await/promises)

Every resolver — `source`/`compute` `fn`, `resolveAttribute`, `resolvePattern` —
is **callback oriented**, matching the push-stream discipline of `@overlay`
adapters. A resolver MUST either:

1. call `cb(err, result)` asynchronously and return `undefined`, or
2. return the result synchronously (pure compute / memoized reads).

`result` may be a value, `{ value, unit, source }`, an interval `{ lower, upper }`,
or `null`/`undefined` (→ uncached `null`). Returning a **Promise is rejected**
with a descriptive error — this library is promise-free. This contract is what
lets the future `value-graph-overlay` bridge implement resolvers over Overlay's
callback/push-stream `FederatedQueryEngine`.

### Black-box nodes

A `blackbox` node is an arbitrary function of its resolved dependencies. Inside
the resolver, `ctx.deps[parent]` is the parent's **value**, and `ctx.entries[parent]`
is the parent's **full result** (`{ value, unit, at, source, fresh }`). Use it to
compose an external query from the parent edges/values — e.g. key an Overlay query
off a `pattern` parent's edge set:

```js
vg.define('risk', {
  operator: 'blackbox',
  parents: ['related_edges', 'base_balance'],
  fn: (subject, params, ctx, cb) => {
    const edges = ctx.deps.related_edges;        // array of edges from the pattern node
    const balance = ctx.entries.base_balance;    // full { value, unit, at, source, fresh }
    overlayQuery(buildAst(subject, edges, balance), cb); // callback-based, no promises
  }
});
```

**OWA fusion operators** (ADR-000): `max`, `min`, `average`, `sum` (capped 1.0),
`sum_unbounded`, `majority`, `median`, `optimistic`, `pessimistic`, `top2`,
`top3`, `priority`, `custom` (user weights), `product`. Numeric values use
position-based weights over descending order; non-numeric values fall back to
`max`/`min`/`majority`.

## Compile → optimize → run (never fused)

- `plan(subject, relation, params)` walks `parents` into a node DAG and OPTIMIZES
  it (compile and optimize are one call; the graph is deliberately **never fused** —
  each operator stays a separately-cacheable node):
  - **Cached-subtree pruning** — a cache-safe (fresh) node becomes a **single
    trivial leaf with no parents**, pruning its dependency subtree, so `run()`
    reads its cache instead of re-evaluating.
  - **Ghost-parent elimination** — a fusion parent that is undeclared AND uncached
    is guaranteed `null`, which fusion filters anyway; it is dropped at compile time
    to avoid a wasted node evaluation. (Not applied to compute/blackbox, which
    receive `null` in deps and may handle it.)
  - Plan metadata: `size`, `recomputeCount` (nodes `run()` evaluates),
    `prunedCount` (nodes served from cache), `ghostPruned` (dropped ghosts).
- `run(plan, cb)` evaluates the DAG in dependency order, dispatching each node to
  its operator and caching the result — all callback-driven, no promises.
  **Parallelism**: all independent parents are fired as soon as they are reached
  (one synchronous pass starts every resolver), so N independent async resolvers
  (e.g. Overlay REST calls) are in flight simultaneously — verified by test
  (max in-flight === N). A node reachable via multiple paths (a diamond/shared
  subtree) is evaluated **exactly once per run**: later paths join the in-flight
  evaluation via waiter fan-out instead of re-firing the resolver (this fixed a
  real double-firing bug for async resolvers).
  **Race safety**: a run only caches its results if no `set`/`invalidate` occurred
  mid-run (run-epoch guard) — a slow read can never clobber an authoritative
  `set`, and a pre-mutation snapshot is never cached after an `invalidate` (both
  fixed + rigor-verified).
- `invalidate(subject, relation, params)` requires the relation — calling it with
  only a subject now throws instead of silently clearing the whole store
  (`invalidate()` with no args still clears everything).
- `invalidate`/`set` compile the **affected dependent DAG** (the relation plus
  every transitive dependent), push `{ stale: true }` triggers DOWN to active
  queries (both notify the affected relation's own watchers; `set` marks the
  set relation's trigger with no `via`, dependents with `via`), and bump
  **per-relation version stamps** — so invalidation is
  **selective**: only the affected closure recomputes; unrelated relations keep
  their cached entries (verified by js-rigor).

## Pull on duplex push

`query(subject, relation, params)` returns a duplex: push `{ get: true }` up
(compile + run, serialized), receive the value down. External sources push
`invalidate`/`set` to send back-channel stale triggers.

## Backing store & larger-than-memory

The graph object itself holds only its **schema** (`relations` — the operator
definitions) and **active streams**; every cached entry lives in the swappable
backing store (`get`/`set`/`delete`/`clear`/`keys`/`deletePrefix`). That is what
makes the value graph **larger-than-memory** — data volume is decoupled from RAM,
exactly the posture of the auth graph (`@arbiter/core`'s condensed typed-array
graph + planned sharded snapshots, ADR-003).

- `createMapStore()` — in-memory default (a `Map`).
- `createFileStore(dir, { maxResident })` — disk-backed reference: one **binary**
  file per entry (the same JSON-free wire encoding the snapshots use), with a
  bounded LRU **hot-page cache** on top. A fresh `ValueGraph` pointed at the same
  directory reads cached values straight from disk without recompute — proven by
  test. **No JSON, no JS bigint** on disk — large integers are `Uint8Array`.
- Any store implementing the synchronous contract is swappable — an mmap-backed
  store, a remote catalog, or (later) an Overlay-backed store.

```js
import { ValueGraph, createFileStore } from '@arbiter/value-graph';

const vg = new ValueGraph({ store: createFileStore('./vg-data', { maxResident: 256 }) });
```

### Compact snapshots, read from disk, transportable

A snapshot is a **single compact, CRC-guarded binary file** — no paths, no
machine-dependent encoding, **no JSON and no JS bigint on the wire** (large
integers are raw `Uint8Array` bytes; params/pattern/objects are deterministic
binary maps) — so a graph moves between processes and machines byte-for-byte.
Two flavours share one wire format:

- **Store snapshot** — opaque `key → entry` dump of the backing store.
  `store.snapshotFile(path)` / `store.restoreFile(path)` on both store flavours;
  `createFileStore(dir, { loadFrom })` constructs a store already loaded from a
  snapshot.
- **Graph snapshot** — the seamless artifact: **schema metadata** (operator,
  parents, weights, TTL, returnType, params — minus functions) **+ every entry**.

```js
vg.snapshotFile('./graph.snap');            // stream to disk, bounded memory
const buf = vg.snapshot();                  // or a portable Buffer
const vg2 = ValueGraph.restore(buf, { resolvers: { balance: () => loadBalance() } });
const vg3 = ValueGraph.loadFile('./graph.snap', { resolvers: { balance: () => loadBalance() } });
```

`restore`/`loadFile` re-declare the schema and load entries with their version
re-stamped, so a restored cache is served **without recompute** (subject to TTL —
old snapshots expire on the wall-clock like any other entry). Functions cannot be
serialized: re-register resolvers via `{ resolvers }`. A corrupt or truncated
snapshot **throws** (magic/version/CRC/parse guards) — it never silently mis-reads.

## Typed values (DSL-declared schema)

The graph carries **schema metadata** from the evidence DSL: `define(rel, { returnType, params })`
records the declared return type (a primitive like `number`/`string`/`boolean`/`buffer`/
`timestamp`/`array`/`interval`, or an entity/Definition name treated as a reference key)
and parameter types. `relationSpec(rel)` introspects it. **There is no JS `bigint`
type** — large integers are `buffer` (`Uint8Array`), matching the JSON-free wire.

- `set(...)` validates the value against `returnType` (untyped relations are
  backward-compatible and skip validation).
- Resolver results are validated against `returnType` too — a wrong-typed resolver
  fails loudly instead of caching garbage.
- **`NaN` is rejected** at both the `set` and resolver-result boundaries — it is
  not a meaningful derived value and would poison downstream OWA.
- `validateValueType(value, type)` is exported for consumers (the DSLValueGraph
  integration uses it for binding + value checks).

This is the storage/typing substrate for `DSLValueGraph` in `@arbiter/evidence-dsl`:
measures declared in the DSL become typed value-graph nodes; attributes that are
neither cached nor computed are stored via `setValue`; `measure()`/`runtime.measure()`
retrieve through the graph.

## Example

```js
import { ValueGraph } from '@arbiter/value-graph';

const vg = new ValueGraph({
  resolveAttribute: (nodeKey, path, params, ctx, cb) => cb(null, graphNodes.get(nodeKey)?.[path])
});

vg.define('base_balance', { operator: 'source', fn: (s, p, ctx, cb) => cb(null, ledgerBalance(s)) });
vg.define('risk_adjusted', { operator: 'compute', parents: ['base_balance'], fn: (s, p, { deps }) => deps.base_balance * 0.9 });
vg.define('effective_clearance', { operator: 'fusion:max', parents: ['base', 'role', 'group'] });
vg.define('avg_score', { operator: 'fusion:custom', parents: ['rep', 'activity', 'verif'], weights: [0.6, 0.3, 0.1] });

const plan = vg.plan('tenant:acme', 'risk_adjusted', {});
vg.run(plan, (err, value) => { /* { value, unit, at, source, fresh } */ });
```

## Deep testing (js-rigor)

`test/value-graph.rigor.test.js` (`npm run test:rigor`) is a js-rigor suite:
- **OWA properties** — `owa`/`weightFor` invariants (sum/avg/max/min/median/top2/
  product/optimistic/pessimistic/custom, string fallbacks) + a `sum` oracle. This
  caught and fixed two real edge cases: `weightFor('top2'|'top3', n)` over-produced
  weight vectors for `n` smaller than the named positions, and `weightFor(op, 0)`
  returned `[1]` for `max`/`min` instead of `[]`.
- **OWA oracle conformance** — an *independent* reimplementation of the ADR-000 OWA
  formula (`refOwaExact`, mirrored arithmetic for bit-exact `===`) confirms every
  operator: exact oracles for sum/max/min/product/median/string fallbacks + capped
  sum, and **arbitrary operator combinations** (a frequency-weighted pick over all
  14 operators × random arrays × random weights/priorities) must match the reference.
- **Fusion DAG oracles** — the graph's fusion path must equal raw `owa` for any
  operator, and **composed DAGs** (fusion-of-fusion, compute→fusion chains) must
  equal the bottom-up reference over the same operators.
- **More correctness oracles** — mixed fusion+compute topologies vs the bottom-up
  reference; **invalidation correctness** (changing a source and invalidating it
  yields the correct new root value across the dependent subtree); **cache-key
  isolation** (subjects never share entries, each computes once); **stale-on-error
  fallback** (a failing resolver after TTL expiry returns the cached value with
  `fresh:false` instead of erroring); **value-type round-trips** (array/interval/
  buffer survive the disk-backed store into a fresh graph instance); **blackbox
  over fusion** (`fusion(a,b)·k` matches the reference); and **selective
  invalidation** (invalidating/setting an unrelated relation leaves its cached
  entry untouched; a sibling independent subtree stays cached).
- **Universal invariants** — for *every* operator: empty→`0`, singleton→element,
  sum/sum_unbounded/product exact, weighted operators bounded within `[min,max]`,
  and monotone non-decreasing under a `+1` bump. Plus: **plan exactness**
  (fully-cached plan is one leaf; invalidation expands to exactly the affected
  closure with unrelated leaves still trivial); **async serialization** (two
  concurrent `{get}` pushes with an async resolver deliver both values and the
  second is served from cache — resolver called once); **params cache-key
  isolation**; **cross-subject invalidation correctness**; **invalidateAll clears
  every subject**; **ghost parents are filtered from fusions**; **attribute/
  pattern hooks receive the documented arguments**; **TTL boundary is exact**
  (fresh inside, stale at the window, `ttl:0` never expires); **`set` preserves
  unit/source**; in-graph string fusion fallbacks + `capSum`; and **duplicate
  parents are a single dependency** (parents are a set).
- **Random-topology DAG oracles** — three mid fusion nodes over randomly-picked
  source pairs plus a root fusion, all with random operators, must match the
  independent bottom-up reference (`refOwaExact`), including the duplicate-parent
  set semantics.
- **Complexity verification** — `run()` on a deep compute chain is `O(n)` resolver
  calls (no recomputation blowup), `plan()` compiles a deep chain in `O(n)`, and
  `weightFor` allocates `O(n)` weights (deterministic cost metrics).
- **Compile-time optimization** — ghost parents are eliminated at compile time
  (semantics preserved for every operator vs `refOwaExact`), a fully-cached plan is
  a single trivial leaf with zero recomputes, and an invalidated plan recomputes
  exactly the affected closure while serving the untouched sibling from cache.
- **Graph caching model (formal conformance)** — `rigor.model.check` runs random
  `get`/`set`/`invalidate`/`mutate` sequences against a reference model of the
  per-relation-version caching contract and the real graph; every value must match.
  Pins the exact semantics: `set` writes fresh without bumping the relation version,
  `invalidate` bumps it and recomputes, and a backing mutation without invalidation
  stays cached until the next invalidation/TTL.
- **Query duplex lifecycle** — a randomized `get`/`invalidate` stream delivers only
  correct values and well-formed stale triggers; `end()` cleans up so post-end
  writes are silent no-ops (including `invalidate` pushes to the closed query); and
  `abort()` surfaces its error on the stream.
- **Store-fault robustness** — a throwing store must never hang or crash the graph:
  a store whose `set` always fails still serves correct values (best-effort uncached
  serve — this fixed a real hang where a run-time `store.set` failure was swallowed
  by the resolver guard), a flaky store recovers (later gets are cached), and a
  store whose `get` fails surfaces the error via callback. The universal OWA
  invariants also pin the constant-array case (all-equal inputs → the value for
  weighted operators).
- **Algebraic OWA invariants** — results are **invariant under input order** (any
  operator: sort-based), and **max/min/average/median are invariant under multiset
  duplication** (each element duplicated; FP-tolerance for average).
- **Params-scoped invalidation & CSE** — invalidating one params set deletes that
  key while every subject/params value stays correct; a **shared subtree** in a
  diamond DAG is computed once across two roots and once across the post-
  invalidation recompute (no redundant evaluation); and `invalidate` clears an
  eager `set` value so the next get recomputes from the resolver.
- **Action-level fault injection** — under injected `rigor.faults` (throw on the
  get action) the callback contract still holds: correct value or surfaced error,
  never a hang.
- **Parallel firing & async CSE** — N independent async resolvers fire
  simultaneously (max in-flight === N), each exactly once; a diamond DAG with
  async resolvers fires its shared node exactly once (no duplicate REST calls);
  and waiter fan-out joins at every level of a deep shared subtree.
- **Stateful protocol, concurrency, edges, benchmark** — a `rigor.object` protocol
  campaign (with `before`/`after`/`between`) pins "after `set(v)` every `get`
  returns `v` until `invalidate`"; **concurrent direct gets** on one graph each
  complete independently and correctly (N calls for N gets, no torn state); a run
  error inside a query duplex is pushed down as `{ error }`; `set` rejects
  unsupported value shapes; resolvers propagate `{ value, unit, source }` onto the
  delivered entry; and a loose **latency benchmark** smoke guards the fast path.
- **Graph semantics** — caching (second get computes nothing), invalidation cascade,
  `set` override, plan pruning, TTL expiry, `blackbox` deps/entries.
- **Callback contract** — handler-based: sync + async resolver delivery, resolver
  error propagation, Promise-return rejection, duplex pull (two gets), and stale
  trigger pushed between values on `invalidate`.
- **Store contract** — model-based conformance (`rigor.model.check`) of
  `createMapStore` and `createFileStore` against a reference `Map` under random
  `set`/`get`/`delete`/`clear`/`deletePrefix` sequences.

## License

ISC