@polyweave/nursery-bench (1.0.3)

Published 2026-07-11 00:26:52 +00:00 by Dvorak

Set up this registry in your project .npmrc file:

@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/

To install the package using npm, run the following command:

npm install @polyweave/nursery-bench@1.0.3

or add it to the package.json file:

"@polyweave/nursery-bench": "1.0.3"

For more information on the npm registry, see the documentation.

nursery-bench

Benchmark suite for nursery protocol scoring with deterministic domains and verifiable ground truth.

Published as @polyweave/nursery-bench.

Synopsis

import {
  createBenchmarkSuite,
  runDomainBenchmark,
  rankProtocols,
  scorecardSummary
} from '@polyweave/nursery-bench';

const suite = createBenchmarkSuite({
  domains: [createHanoiDomain(), createJsonPatchDomain(), createBtSpecRepairDomain()],
  protocols: generateProtocolCandidates(10)
});

const results = await runDomainBenchmark(suite);
console.log(scorecardSummary(results));

Install

npm install @polyweave/nursery-bench

Description

The nursery-bench package provides a benchmark suite for evaluating Polyweave nursery protocol scoring against deterministic domains with verifiable ground truth. Unlike real-world tasks where evaluation is inherently approximate, the benchmark domains have well-defined inputs, outputs, and scoring criteria — allowing precise calibration of scoring profiles and protocol quality.

Pre-built domains include: Tower of Hanoi (algorithmic planning with known optimal solutions), JSON Patch (deterministic structural transformations), BT Spec Repair (behavior tree validation with ground-truth fixes), Fact Graph Query (structured graph queries with known answers), and HumanEval Mini (code generation with test-driven verification).

The package also provides a simulated runner for deterministic execution, protocol candidate generation, and a scorecard system for comparing predicted vs. actual protocol quality.

API

`createBenchmarkSuite(options)`

Creates a benchmark suite combining domains and protocols.

Parameters:

options.domains — Array of domain objects with { name, generateInstances, validate, groundTruth }.
options.protocols — Array of BT spec strings to evaluate.
options.scoringProfile — Scoring weight profile for protocol evaluation.
options.simulator — Custom simulator (defaults to createSimulatedRunner).

Returns: { domains, protocols, scoringProfile, simulator }

`runDomainBenchmark(suite)`

Runs all protocols against all domains and collects results.

Parameters:

suite — Benchmark suite from createBenchmarkSuite.

Returns: { results: [{domain, protocol, score, objectiveScores, groundTruthMatch}], aggregate }

`rankProtocols(results)`

Ranks protocols by aggregate score across all domains.

Returns: [{ protocol, totalScore, rank, scores: [{domain, score}] }]

`scorecardSummary(results)`

Generates a human-readable summary of benchmark results.

Returns: { bestProtocol, domainScores, protocolRanks, calibration }

Domain Constructors

createHanoiDomain() — Tower of Hanoi planning domain.
createJsonPatchDomain() — JSON Patch transformation domain.
createBtSpecRepairDomain() — BT spec repair validation domain.
createFactGraphQueryDomain() — Fact graph structured query domain.
createHumanEvalMiniDomain() — Code generation with test verification.

Each returns: { name, generateInstances, validate, groundTruth }.

`generateProtocolCandidates(count, options)`

Generates diverse protocol candidate specs for benchmarking.

`createSimulatedRunner(options)`

Creates a deterministic simulated execution runner.

`computeScorecard(results)`

Computes a detailed scorecard from benchmark results.

`comparePredictedVsActual(results)`

Compares nursery-predicted scores against actual benchmark ground-truth scores.

Examples

// Full benchmark pipeline
const suite = createBenchmarkSuite({
  domains: [
    createHanoiDomain(),
    createJsonPatchDomain()
  ],
  protocols: generateProtocolCandidates(20)
});

const results = await runDomainBenchmark(suite);
const ranked = rankProtocols(results);

console.log('Top protocol:', ranked[0].protocol);
console.log('Score:', ranked[0].totalScore);

// Calibration check
const { calibration } = scorecardSummary(results);
if (calibration.error > 0.1) {
  console.log('Scoring profile needs tuning — high prediction error');
}

Tests

npm test

Tests cover benchmark suite creation, domain ground-truth verification, protocol ranking across domains, simulated runner determinism, scorecard computation, predicted-vs-actual comparison, and each pre-built domain's instance generation and validation.

Requirements

Node.js 18 or later. ESM only.

Caveats

The simulated runner is deterministic and does not reflect real-world LLM variability. Actual protocol performance may differ.
Pre-built domains are designed for calibration, not comprehensive coverage. Add custom domains for production validation.
Protocol candidate generation uses stochastic methods; seed control is recommended for reproducible benchmarks.

License

MIT

Dependencies

ID	Version
@polyweave/bt-harness	*
@polyweave/core	*
@polyweave/protocol-metrics	*
@polyweave/redflag	*
@polyweave/sampler	*
@polyweave/state	*

Development Dependencies

ID	Version
@rigor/core	*

Peer Dependencies

ID	Version
@polyweave/deepseek	*
@polyweave/nursery	*

polyweave nursery benchmark protocol scoring

Details

npm

2026-07-11 00:26:52 +00:00

John Dvorak

SEE LICENSE IN LICENSE

16 KiB

Assets (1)

nursery-bench-1.0.3.tgz 16 KiB

Versions (3) View all

1.0.6

2026-07-11

1.0.3