Polyweave

Repositories

1

Projects Packages

@polyweave/nursery-bench (1.0.6)

Published 2026-07-11 15:28:00 +00:00 by Dvorak

Installation

Set up this registry in your project .npmrc file:

@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/

To install the package using npm, run the following command:

npm install @polyweave/nursery-bench@1.0.6

or add it to the package.json file:

"@polyweave/nursery-bench": "1.0.6"

For more information on the npm registry, see the documentation.

About this package

@polyweave/nursery-bench

Benchmark suite for nursery protocol scoring with deterministic domains and verifiable ground truth. Evaluates how well nursery-predicted scores match actual protocol quality.

Synopsis

import { createBenchmarkSuite, runDomainBenchmark, rankProtocols, scorecardSummary } from '@polyweave/nursery-bench';
import { createHanoiDomain, createJsonPatchDomain } from '@polyweave/nursery-bench';
import { generateProtocolCandidates, createSimulatedRunner } from '@polyweave/nursery-bench';

// Full pipeline
const domains = [createHanoiDomain(), createJsonPatchDomain()];
const protocols = generateProtocolCandidates({ count: 12 });
const runner = createSimulatedRunner();

const result = runDomainBenchmark(protocols, domains[0], runner);
console.log('Best protocol:', result.best.protocolId, 'p1:', result.best.p1Actual);
console.log('Correlation (claimed vs actual):', result.claimedVsActualRho);

// Rank across domains
const ranked = rankProtocols(result.scorecards);
console.log(scorecardSummary(result));

// Suite mode
const suite = createBenchmarkSuite({ domains, protocols });
const allResults = suite.runAll();

Install

npm install @polyweave/nursery-bench

Why

Nursery protocol scoring uses claimed metrics (p0, c, gamma, validRate from the protocol design) to compute predicted p1 and reliability-economic scores. Before trusting these scores for production protocol growth, you need to verify they correlate with actual performance. This package provides deterministic benchmark domains with known-optimal solutions, a simulated runner that injects protocol-claimed error rates as ground truth, and a scorecard system comparing nursery predictions against simulated actuals — all without requiring live LLM API calls.

API

createBenchmarkSuite(options)

Creates a benchmark suite that combines domains, protocols, and a simulated runner.

Parameters:

Name	Type	Required	Default	Description
options.protocols	array	No	generateProtocolCandidates(options)	Array of protocol candidate objects
options.hanoi	object	No	{}	Options forwarded to createHanoiDomain
options.jsonPatch	object	No	{}	Options forwarded to createJsonPatchDomain
options.btSpecRepair	object	No	{}	Options forwarded to createBtSpecRepairDomain
options.factGraphQuery	object	No	{}	Options forwarded to createFactGraphQueryDomain
options.humanEvalMini	object	No	{}	Options forwarded to createHumanEvalMiniDomain
options.useDomainSpec	boolean	No	false	If true, use domain-spec-adapter instead of direct constructors
options.domainNames	array	No	—	Domain names to include when useDomainSpec
options.trainCount	number	No	80	Training instances count
options.holdoutCount	number	No	40	Holdout (test) instances count

Return value:

{
  runAll(overrides) → { domainCount, protocolCount, results, summary },
  runDomain(domainName, overrides) → result (throws if domain unknown),
  getDomains() → [domain],
  getProtocols() → [protocol]
}

runDomainBenchmark(protocols, domain, runner, overrides)

Runs all protocols against a single domain and collects nursery-predicted-vs-actual comparisons.

Parameters:

Name	Type	Required	Default	Description
protocols	array	Yes	—	Array of protocol candidate objects
domain	object	Yes	—	Domain object with { name, train, holdout, verifier, slices }
runner	function	Yes	—	Simulated runner function (from createSimulatedRunner)
overrides	object	No	—	Passed through to runner, not used directly

Return value:

{
  domain, description, calibrationCount, holdoutCount, protocolCount,
  scorecards: [{ protocolId, domain, instances, p0, p1Actual, correctionRate,
                 corruptionRate, validRate, redflagCount, verifiedCount,
                 preCorrectCount, postCorrectCount, totalTokens, costPerVerified,
                 modelCost, breakEvenCorrection, channel,
                 nurseryP1Predicted, nurseryExpectedGain, nurseryScore,
                 nurseryEconScore, nurseryTScore, nurseryPredictionError, economics }],
  claimedVsActualRho: Spearman's rho between nurseryP1Predicted and p1Actual,
  nurseryScoreVsActualRho: Spearman's rho between nurseryScore and p1Actual,
  sliceAnalysis: { bestProtocol, phasePlane, gatingDecisions, activeSlices, suppressedSlices } | null,
  best: best scorecard by p1Actual,
  mostEfficient: best scorecard by costPerVerified,
  verdict: 'EXCELLENT' | 'GOOD' | 'ADEQUATE' | 'WEAK' (based on Spearman thresholds)
}

rankProtocols(scorecards, options)

Ranks protocols by a chosen metric across domains.

Parameters:

Name	Type	Required	Default	Description
scorecards	array	Yes	—	Scorecard array from runDomainBenchmark
options.metric	string	No	'p1Actual'	Metric to rank by ('p1Actual', 'nurseryScore', etc.)

Return value: Array of { rank, protocolId, domain, p1Actual, nurseryP1Predicted, nurseryScore, costPerVerified } sorted descending by metric.

scorecardSummary(result)

Generates a human-readable summary of benchmark results.

Return value: { domain, best, bestP1, claimedVsActualRho, nurseryScoreVsActualRho, totalProtocols, verdict }

Domain Constructors

All five domain constructors return objects with { name, description, train, holdout, slices, verifier, sliceFn, taskFn }.

createHanoiDomain(options)

Tower of Hanoi planning domain.

Option	Type	Default	Description
trainCount	number	80	Number of training instances
holdoutCount	number	40	Number of holdout instances
diskRange	array	[3, 6]	Range of disk counts [min, max]

Slices: shallow (disks ≤ 3), medium (disks ≤ 5), deep (disks > 5). Each instance has { id, slice, disks, state, goal, validMoves, correctMove, task }. Verifier checks disk index bounds, peg validity, source correctness, and valid move rules.

createJsonPatchDomain(options)

JSON Patch (RFC 6902) transformation domain.

Option	Type	Default	Description
trainCount	number	60	Training instances
holdoutCount	number	30	Holdout instances

Slices: simple, nested, complex. Each instance has { id, slice, op, field, source, target, correctPatch, task }. Verifier checks valid patch array, applies patch to source, and compares result to target.

createBtSpecRepairDomain(options)

BT spec repair validation domain.

Option	Type	Default	Description
count	number	40	Total instances (60% train, 40% holdout)

Slices: syntax, semantic, structural. Four bug types: missing action definition, unbalanced parens, invalid postcondition operator (= instead of ==), missing TREE section. Verifier checks fixed: true and required BT spec sections.

createFactGraphQueryDomain(options)

Fact graph query generation domain.

Option	Type	Default	Description
count	number	40	Total instances

Slices: simple, join, multi. Each instance specifies entity, relation, and expected FACT_QUERY structure. Verifier checks correct type: 'FACT_QUERY' and content entity/relation fields.

createHumanEvalMiniDomain(options)

Code generation with test-driven verification.

Option	Type	Default	Description
count	number	30	Total instances

Slices: easy, medium, hard. Five problems: is_even, add_two, string_reverse, list_max, fibonacci. Verifier evaluates code via new Function() and runs test cases comparing output to expected values.

generateProtocolCandidates(options)

Generates diverse protocol candidate specs for benchmarking using 12 structural genes.

Parameters:

Name	Type	Default	Description
options.count	number	12	Number of candidates (max 12, limited to available genes)

Twelve gene types with varying combinations of verifier, redflag, quorum, tier (cheap/mid/expensive), decomposition, action count, and check count. Each gene maps to: fully-built BT spec text, estimated metrics (p0, gamma, validRate, avgVotes), tier-based token economics (input/output tokens and costs), uncertainty signals, self-improving signals, temperature settings, redflag configuration, and oracle headroom. Returns array of protocol candidate objects compatible with scoreProtocolCandidate.

createSimulatedRunner(options)

Creates a deterministic simulated execution runner that injects protocol-claimed error rates as ground truth.

How it works: For each instance, the runner:

1. Uses a deterministic hash-based Bernoulli trial for "before correct" (using protocol's p0)
2. Generates domain-aware correct or incorrect output based on the trial
3. If usesRedflag, samples from redflagRate and resamples on flag
4. If usesVerifier, runs the domain's verifier function on the output
5. If no verifier, applies corruption rate to determine "after correct"
6. Tracks (n00, n01, n10, n11) counts for correction/corruption channel estimation
7. Computes token cost from protocol's tokenEconomics

Return value (runner function signature): (protocol, domain, overrides) → { protocolId, domain, instanceCount, preCorrectCount, postCorrectCount, redflagCount, verifiedCount, totalTokens, p0, p1, delta, validRate, modelCostUsd, counts }

computeScorecard(protocol, simResult)

Computes a detailed scorecard from a protocol and its simulated run result.

Return value: { protocolId, domain, instances, p0, p1Actual, correctionRate, corruptionRate, validRate, redflagCount, verifiedCount, preCorrectCount, postCorrectCount, totalTokens, costPerVerified, modelCost, breakEvenCorrection, channel }

channel is computed via estimateChannel from @polyweave/protocol-metrics using the (n00, n01, n10, n11) counts. costPerVerified = modelCost / postCorrectCount (or Infinity if zero). breakEvenCorrection = (p0 / (1-p0)) × gamma (or Infinity if p0 >= 1).

comparePredictedVsActual(scorecards, options)

Compares nursery-predicted p1 values against actual benchmark p1 values.

Return value: Array of { protocolId, domain, actualP1, nurseryPredictedP1, nurseryPredictionError, correctionRate, corruptionRate, costPerVerified, nurseryScore }

Error Conditions

Condition	Where	Error
Unknown domain name	runDomain(domainName, ...)	throw new Error('Unknown domain: ' + domainName)
Instance verifier throws	Simulated runner	Caught, sets verdict to { passed: false, code: 'EXCEPTION' }

Scorecard Verdicts

The suite computes Spearman rank correlation between:

1. claimedVsActualRho: Correlation between nurseryP1Predicted and p1Actual
2. nurseryScoreVsActualRho: Correlation between nurseryScore and p1Actual

Verdict thresholds:

• EXCELLENT: claimed > 0.9 && nursery > 0.7
• GOOD: claimed > 0.8
• ADEQUATE: claimed > 0.5
• WEAK: otherwise

Tests

npm test

Runs node --test test/*.test.js. Covers:

• createBenchmarkSuite construction with custom domains and protocols
• Domain ground-truth verification (hanoi move validity, json patch correctness)
• Protocol ranking across all 5 domains
• createSimulatedRunner determinism (same seed → same results)
• computeScorecard channel estimation and cost calculations
• comparePredictedVsActual prediction error computation
• scorecardSummary verdict classification
• Each pre-built domain's instance generation, verifier correctness, and slice classification

Requirements

• Node.js >= 22.0.0
• ESM only ("type": "module")
• Dependencies: @polyweave/bt-harness, @polyweave/core, @polyweave/protocol-metrics, @polyweave/redflag, @polyweave/sampler, @polyweave/state
• Peer dependencies: @polyweave/deepseek, @polyweave/nursery

Dependencies

Dependencies

ID	Version
@polyweave/core	*
@polyweave/protocol-metrics	*
@polyweave/redflag	*
@polyweave/sampler	*

Development Dependencies

ID	Version
@rigor/core	*

Peer Dependencies

ID	Version
@polyweave/deepseek	*
@polyweave/nursery	*

Keywords

polyweave nursery benchmark protocol scoring

Details

npm

2026-07-11 15:28:00 +00:00

4

John Dvorak

SEE LICENSE IN LICENSE

latest

18 KiB

Assets (1)

nursery-bench-1.0.6.tgz 18 KiB

Versions (3) View all

1.0.6 2026-07-11

1.0.3 2026-07-11

1.0.2 2026-07-10