@polyweave/state (1.0.3)
Installation
@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/npm install @polyweave/state@1.0.3"@polyweave/state": "1.0.3"About this package
@polyweave/state
Transactionally versioned state stores with DVVSet-based conflict detection, lazy MVCC snapshot isolation, and kind-driven entity projections with declared relations.
Synopsis
import { createMultiStateStore, createDocumentStore, createTextStore, createTreeStore, createFactGraph } from '@polyweave/state';
var store = createMultiStateStore({ name: 'main' });
var docs = store.createDocumentStore({ name: 'docs' });
docs.sink.write({ type: 'DOC_UPDATE', content: { id: 'a', text: 'Hello world' } });
var result = docs.get('a');
Install
npm install @polyweave/state
Architecture
All stores share a push-stream protocol: write frames via sink.write(frame), subscribe to output via source.pipe(listener).
Internal mutation is synchronous and single-threaded — there is no actual concurrency, only logical races detected
through version vectors.
Version Vectors (DVVSet)
Every versioned unit (fact, section, tree path, document) carries a dots object { txnId: counter }. Dots are
advanced on transactional writes only; non-txn writes do not increment them.
Shared primitives in src/lib/dvv-set.js:
dotsDominate(a, b) → a strictly dominates b: a includes b and has additional causal knowledge
dotsEqual(a, b) → a == b
dotsConflicting(a, b) → neither a ≥ b nor b ≥ a (fork)
advanceDot(dots, id) → dots[id] = (dots[id] || 0) + 1
A zero-allocation FlatDots variant (Uint32Array) is available for hot paths.
Conflict Modes
| Mode | Behavior | Stores |
|---|---|---|
| Persistent | Conflicting value stored alongside committed value. Multiple versions visible. System continues. Resolution is higher-order. | Fact Graph, Document Store, Tree Store |
| Abortive | Transaction aborts. Key appended to blocked set. All writes (including non-txn) rejected until resolved. | Text Store, Cluster Tree Store, Vector Index |
| Delegated | Conflict detection handled by the individual stores wrapped by the federation layer. | Multi-State Store |
Conflict frames standardized via src/lib/conflict-frames.js: emitTxnConflict, emitKeyBlocked, emitKeyUnblocked, resolveConflict.
MVCC Snapshot Isolation
Fact Graph, Document Store, and Tree Store support lazy shadow chain MVCC via src/lib/mvcc.js. When any snapshot
transaction is open, all commits go to a shadow chain instead of mutating the base state in-place. When the last
snapshot closes, the chain condenses into the base. Snapshot transactions see state frozen at their start time;
read-committed and non-txn queries see the full merged chain. Shadow commits are WAL-logged for crash recovery.
Stores (8)
1. Fact Graph
createFactGraph(options?)
A fact-based triple store (entity, relation, value) with temporal and spatial indexing, composite index sketch,
and streaming lazy query.
| Write frames | FACT_INSERT, FACT_UPDATE, FACT_DELETE |
|---|---|
| Conflict mode | persistent |
| MVCC | snapshot isolation |
| Key API | getFact(factId), getFactVersion(factId, chainLimit?), getMergedDots(factId, chainLimit?), compareAndSwap(factId, dots, value), checkConflicts(txnId), getConflicts({ factId }), resolveConflict(factId, resolution), searchStream(query, options), query(dsl), killSnapshot(txnId), getMvccState() |
2. Document Store
createDocumentStore(options?)
A section-based document store. Documents are composed of sections identified by sectionId. Supports streaming
query and persistence integration.
| Write frames | DOC_UPDATE |
|---|---|
| Conflict mode | persistent |
| MVCC | snapshot isolation |
| Key API | getSection(sectionId), getSectionVersion(sectionId, chainLimit?), getMergedDots(sectionId, chainLimit?), compareAndSwap(sectionId, dots, value), checkConflicts(txnId), getConflicts({ sectionId }), resolveConflict(sectionId, resolution), querySectionsStream(query, options) |
3. Tree Store
createTreeStore(options?)
A wormhole-trie-backed tree store. Paths are the versioned unit. Rearranging the tree structure IS versioned data.
| Write frames | TREE_UPSERT, TREE_LINK, TREE_UNLINK |
|---|---|
| Conflict mode | persistent |
| MVCC | snapshot isolation |
| Key API | getCandidates(query, options), getPathVersion(treeId, path, chainLimit?), getMergedDots(treeId, path, chainLimit?), compareAndSwapPath(treeId, path, dots, node), checkConflicts(txnId), getConflicts({ treeId, path }), resolveConflict(treeId, path, resolution), searchStream(query, options) |
4. Text Store
createTextStore(options?)
Full-text search with BM25 scoring, inverted index, and optional embedding support. Two-phase commit: Phase 1 dry-runs all operations against committed dots, Phase 2 applies. No partial inverted index corruption.
| Write frames | TEXT_INDEX, TEXT_REMOVE |
|---|---|
| Conflict mode | abortive |
| MVCC | no (two-phase commit, doc-level CAS) |
| Key API | index(docId, text, options?), remove(docId), search(query, options?), searchStream(query, options), getDocumentVersion(docId), checkConflicts(txnId), isBlocked(docId), resolveConflict(docId, resolution), compareAndSwapDocument(docId, dots, value) |
5. Cluster Tree Store
createClusterTreeStore(options?)
Quality-threshold (QT) clustering over vector embeddings. Maintains a cohesive cluster tree with auto-rebuild on drift.
| Write frames | CLUSTER_UPSERT |
|---|---|
| Conflict mode | abortive |
| MVCC | no (doc-level CAS) |
| Key API | upsert(docId, embedding, metadata?), remove(docId), buildTree(), searchStream(query, options), isBlocked(docId), resolveConflict(docId, resolution), getEmbeddingVersion(docId) |
6. Text Embedding Vector Store
createTextEmbeddingVectorStore(textStore)
A thin read-only adapter that wraps a Text Store, exposing a vector-index-like API surface. No state or conflict detection of its own.
| Write frames | N/A (passes through to Text Store) |
|---|---|
| Conflict mode | N/A |
| MVCC | N/A |
| Key API | getCandidates(query), getSignals(), stats(), supportsDocMask() |
7. Vector Index
createVectorIndex(options?)
A low-level multi-algorithm vector index (JAG, RNG, KDT, BKT) with product quantization, paged sampling, and row-level CAS-based conflict detection.
| Write frames | N/A (direct method calls: addBatch, deleteId) |
|---|---|
| Conflict mode | abortive (return-value conflicts) |
| MVCC | no (row-level CAS) |
| Key API | addBatch(rows), deleteId(id), search(query, k?), build(), getRowVersion(id), isBlocked(id), resolveConflict(id, resolution), unblockAll() |
8. Multi-State Store
createMultiStateStore({ textStore, treeStore, graphStore, docStore, projections?, ... })
A federation store that wraps text, tree, graph, and document stores for parallel retrieval. Delegates writes to the appropriate underlying store. Provides unified lazy query with variable binding, multihop traversal, cross-store candidate aggregation, weighted scoring, filter resolution, and entity projections with kind-driven fields and declared relations.
Supports multi-store transactions — atomic commit across all stores with a shared txnId. Snapshot isolation
forwarded to MVCC-capable stores; abortive-dominant atomicity.
Hybrid Memory + Disk Offload (recommended)
For balanced memory usage, keep graph/tree indexes in memory and offload heavy text/document payloads via file-backed WAL durability.
import { FileWALStore } from '@polyweave/core';
import {
createDocumentStore,
createFactGraph,
createMultiStateStore,
createTextStore,
createTreeStore
} from '@polyweave/state';
const walStore = new FileWALStore('./data/polyweave');
walStore.initialize();
const docStore = createDocumentStore({
durability: { enabled: true, wrapperId: 'docs', walStore }
});
const textStore = createTextStore({
// Keep BM25 index in memory, offload raw document text to document-store
storeText: false,
documentStore: docStore,
persistToDocumentStore: true,
lazyLoadDocumentText: true,
durability: { enabled: true, wrapperId: 'text', walStore }
});
const ms = createMultiStateStore({
textStore,
docStore,
graphStore: createFactGraph(), // in-memory hot index
treeStore: createTreeStore(), // in-memory topology
durability: { enabled: true, wrapperId: 'multi', walStore }
});
| Write frames | All store frames + TXN_BEGIN, TXN_COMMIT, TXN_ABORT, KILL_SNAPSHOT, KILL_ALL_SNAPSHOTS, DESCRIBE |
|---|---|
| Conflict mode | Delegated (abortive-dominant across stores) |
| MVCC | Snapshot forwarded to MVCC stores; chainIndices captured per store |
| Key API | search(query, options?), query(query, options), queryStream(query, options), multihopQueryStream(query, options), getAttributes(docId), getState(), getCursorRegistry(), getMvccState(), getMultiTxnState(txnId), triggerRestore() |
Entity Projections
Entity projections let you define entity types as collections of fields and relations. The Multi-State Store derives which stores to create, which frames to emit, and how to read data back. The caller never names a store — they name what the field means.
Defining a Projection
const ms = createMultiStateStore({
// Pre-created stores (optional — auto-created from kinds if omitted)
graphStore: createFactGraph(),
textStore: createTextStore(),
projections: {
Author: {
primaryKey: 'docId',
fields: {
name: 'value',
bio: 'content',
},
relations: {
authored: { to: 'Article', inverse: 'authors' }
}
},
Article: {
primaryKey: 'docId',
fields: {
title: 'value',
body: 'content',
created: 'timestamp',
vec: 'embedding:768',
},
relations: {
authors: { to: 'Author', inverse: 'authored', many: true }
}
}
}
});
Field Kinds
Each field declares a kind — a shorthand string or { kind, ... } object. The system derives store routing
and frame construction automatically.
| Kind | Stores | Description |
|---|---|---|
value |
graph | String attribute (fact: entity=$docId, relation=$field) |
value:n |
graph | Number attribute |
value:b |
graph | Boolean attribute |
value:? |
graph | Nullable string |
content |
document+text | Document section + text index |
content:stored |
document | Document section only (no search) |
content:index |
text | Text index only (searchable, not in doc) |
location:/root |
tree | Tree path, auto-set on insert |
location:/root/link |
tree | Directional TREE_LINK |
embedding:768 |
vector+cluster | Vector (768d) + auto-clustered |
embedding:768:tag |
vector+cluster | Per-tag vector + cluster |
embedding:768:tag:rng |
vector+cluster | With RNG algorithm |
ref:Entity |
graph | Single entity reference |
refs:Entity |
graph | Multi-entity references |
timestamp |
graph | Auto-set ISO timestamp on insert |
timestamp:update / ts:update |
graph | Auto-set on insert + update |
geometry:point |
graph | GeoJSON Point coordinates |
expiry:3600 |
graph | Auto-set TTL (seconds) |
Field overrides via object form:
fields: {
author: { kind: 'value', relation: 'creator' },
body: { kind: 'content', bm25: { k1: 0.8 } },
path: { kind: 'location', root: '/articles', template: '/articles/$docId' },
emb: { kind: 'embedding', dim: 768, tag: 'img', algorithm: 'rng' },
created: { kind: 'timestamp', relation: 'published_at' },
}
Relations
Relations are named predicates between entity types, distinct from fields:
| Property | Description |
|---|---|
to |
Target entity type |
inverse |
Inverse predicate name on the target |
many |
true (default) for array values, false for single value |
relations: {
// Shorthand — just the target type name
authored: 'Article',
// Full spec with inverse and cardinality
authors: { to: 'Author', inverse: 'authored', many: true },
}
CRUD API
// Insert — atomic across all stores
const article = ms.insert('Article', {
docId: 'art:1',
title: 'Machine Learning',
body: 'An introduction...',
authors: ['auth:1', 'auth:2']
});
// Get — raw values
const raw = ms.get('Article', 'art:1');
// { docId: 'art:1', title: '...', body: '...', authors: ['auth:1', 'auth:2'], _dots: {...}, _sources: {...} }
// Get with relation expansion
const expanded = ms.get('Article', 'art:1', { expand: ['authors'] });
// { docId: 'art:1', title: '...', authors: [{ docId: 'auth:1', name: 'Alice' }, ...] }
// Expand all relations
const full = ms.get('Article', 'art:1', { expand: '*' });
// Update — replaces only specified fields
ms.update('Article', 'art:1', { title: 'Deep Learning' });
// Delete — removes from all stores + relation facts
ms.delete('Article', 'art:1');
// Search with expansion
const hits = ms.searchEntities('Article', { text: 'learning' }, { expand: ['authors'] });
Validation
At projection registration time, the system validates:
- Field kinds — must be a recognized kind string or
{ kind, ... }object - Relation targets — must reference a registered projection
- Inverse consistency — inverse predicate must exist on the target entity
- Name collisions — field name must not duplicate a relation name on the same entity
Auto-Store Creation
When projections are provided without pre-created stores, the MS auto-creates:
- graph (fact graph)
- document (document store)
- text (text store)
- tree (tree store)
- cluster:* (cluster tree per tag)
Vector stores require pre-built datasets and are NOT auto-created. Pre-created stores take precedence.
Store Contract
DESCRIBE Protocol
All stores handle the DESCRIBE frame, returning a DESCRIPTION frame with a lightweight state summary.
The Multi-State Store aggregates across all registered stores.
// Frame protocol (works on any store or MS)
store.sink.write({ type: 'DESCRIBE', content: { streamId: 'req1' } });
// → DESCRIPTION { content: { storeType, ...counts, mvcc, conflicts, blocked } }
// Direct API
store.describe() → { storeType, factCount, transactionCount, mvcc, ... }
ms.describe() → { storeType: 'multiState', stores: { graph: {...}, text: {...} }, ... }
Transaction Contract
| Method | Signature | Purpose |
|---|---|---|
checkConflicts(txnId) |
→ { conflicts, mode } |
Non-destructive pre-flight conflict check |
getXxxVersion(key, chainLimit?) |
→ dots |
Current dots for a key |
getMergedDots(key, chainLimit?) |
→ dots |
Merged dots (base + shadow chain, MVCC stores) |
isBlocked(key) |
→ boolean |
Whether a key is locked (abortive stores) |
killSnapshot(txnId) |
→ boolean |
Kill a snapshot transaction (MVCC stores) |
describe() |
→ description object |
Lightweight state summary |
Transaction Protocol
All stores (except Vector Index) follow the standard transaction protocol:
store.sink.write({ type: 'TXN_BEGIN', content: { txnId } });
// read + write operations with txnId...
store.sink.write({ type: 'TXN_COMMIT', content: { txnId } });
// or
store.sink.write({ type: 'TXN_ABORT', content: { txnId } });
Snapshot transactions on MVCC-enabled stores:
store.sink.write({ type: 'TXN_BEGIN', content: { txnId, isolation: 'snapshot' } });
Conflict detection is per-key: only keys that were both read AND written by a transaction are checked. Writes to unread keys skip conflict detection (last-writer-wins).
Multi-Store Transactions
const ms = createMultiStateStore({ graphStore: fg, textStore: ts, docStore: ds });
// Atomic write across 3 stores
ms.sink.write({ type: 'TXN_BEGIN', content: { txnId: 't1', isolation: 'snapshot' } });
ms.sink.write({ type: 'FACT_INSERT', content: { factId: 'f1', ..., txnId: 't1', targetStore: 'graph' } });
ms.sink.write({ type: 'TEXT_INDEX', content: { docId: 'd1', text: '...', txnId: 't1', targetStore: 'text' } });
ms.sink.write({ type: 'DOC_UPDATE', content: { sectionId: 's1', text: '...', txnId: 't1', targetStore: 'document' } });
ms.sink.write({ type: 'TXN_COMMIT', content: { txnId: 't1' } });
// All stores committed with dots { t1: 1 } — or all aborted if any abortive conflict
// Abortive-dominant: a text store conflict aborts the fact graph write too
ms.sink.write({ type: 'TXN_BEGIN', content: { txnId: 't2' } });
ms.sink.write({ type: 'TEXT_GET', content: { docId: 'd1', txnId: 't2', targetStore: 'text' } });
// ... external write to d1 ...
ms.sink.write({ type: 'TEXT_INDEX', content: { docId: 'd1', text: '...', txnId: 't2', targetStore: 'text' } });
ms.sink.write({ type: 'FACT_INSERT', content: { factId: 'f2', ..., txnId: 't2', targetStore: 'graph' } });
ms.sink.write({ type: 'TXN_COMMIT', content: { txnId: 't2' } });
// Text store conflict → TXN_CONFLICT(abortive), d1 blocked, f2 never written
// Kill snapshot across all MVCC stores
ms.sink.write({ type: 'KILL_SNAPSHOT', content: { txnId: 't1' } });
ms.sink.write({ type: 'KILL_ALL_SNAPSHOTS', content: {} });
// Aggregate MVCC state
ms.getMvccState(); // { hasActiveShadows, openCount, chainSize, openTxnIds, byStore }
ms.getMultiTxnState('t1'); // { txnId, isolation, subTxns, chainIndices, startedAt }
Library Modules
| Module | Description |
|---|---|
src/lib/dvv-set.js |
DVVSet version vector primitives (zero-allocation + FlatDots Uint32Array variant) |
src/lib/mvcc.js |
Lazy MVCC shadow chain (beginSnapshot, endSnapshot, commitToShadow, condense) |
src/lib/conflict-frames.js |
Standard conflict frame factories |
src/lib/kind-registry.js |
Kind normalization + store detection from field kind specs |
src/state/multi-state/entity-projection.js |
Entity projection subsystem (fields + relations, CRUD, expansion) |
Testing
# All state tests
node --test test/*.test.js # 553 tests
# Entity projection tests (65 tests)
node --test test/entity-projection.test.js
node --test test/entity-projection.property.test.js
node --test test/entity-relation.test.js
# Multi-state store tests (19 tests)
node --test test/multi-state-store.test.js
# Store-specific tests
node --test test/fact-graph-conflict.test.js
node --test test/mvcc.test.js
node --test test/mvcc.property.test.js
Design Decisions
- In-place commit is safe: JavaScript is single-threaded and synchronous at
sink.write(). No concurrent mutation exists. - Kind-driven store selection: The caller declares what a field means (
'value','content','embedding:768'); the system chooses stores, frame types, and read strategies. Never name a store. - Relations are first-class: Relations between entities are declared separately from fields, with named predicates, inverses, and cardinality. They're stored as fact graph facts and expanded on read.
- Tree-store path-level dots: The versioned unit is
treeId||pathKey, notnodeId, because rearranging the tree is versioned data. - Text-store two-phase commit: Phase 1 checks all ops against committed dots; Phase 2 applies atomically.
- MVCC shadow chain over clone: Zero-allocation shadow chain. No clone cost for reads when the chain is empty or the query path is unaffected.
- Conflicts are per-key: Only keys that were read AND written trigger conflict checks.
- Dual API: Every store exposes both a direct API (
store.getFact(),store.describe()) and a frame protocol (sink.write,source.pipe). Both are first-class; the frame protocol enables push-stream composition. - Federated compatible: All stores are
{ source, sink }duplexes following the push-stream contract.
Tests
npm test
64 tests covering DVVSet conflict detection, lazy MVCC snapshot isolation, transactional commit/rollback/abort, conflict resolution, kind-driven entity projections, streaming cursors, and store-level property tests.
Requirements
Node.js 18 or later. ESM only.
See Also
- BT Harness — Uses state stores for blackboard
- Encyclopedia — Multi-state federation
- Core Frames API — Frame types and protocol
License
MIT
Dependencies
Dependencies
| ID | Version |
|---|---|
| heapify | ^0.6.0 |
| porter-stemmer | ^0.9.1 |
Development Dependencies
| ID | Version |
|---|---|
| @rigor/core | * |
Peer Dependencies
| ID | Version |
|---|---|
| @polyweave/core | * |
| @polyweave/durability | * |