Polyweave
Repositories
1
 Projects Packages

@polyweave/protocol-metrics (1.0.2)

Published 2026-07-11 00:55:55 +00:00 by Dvorak

Installation

@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/
npm install @polyweave/protocol-metrics@1.0.2
"@polyweave/protocol-metrics": "1.0.2"

About this package

ProtocolMetrics

Correction/corruption telemetry and gating primitives for LLM protocols.

Synopsis

import { createProtocolMetrics, estimateChannel, phasePlaneDiagnosis } from '@polyweave/protocol-metrics';

const metrics = createProtocolMetrics();

metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: false, afterCorrect: true });
metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: false, afterCorrect: true });
metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: true, afterCorrect: true });

const channel = metrics.estimate('qa-step', 'easy');
// channel.c = correction rate (wrong → right)
// channel.gamma = corruption rate (right → wrong)
// channel.delta = net accuracy change

const diagnosis = phasePlaneDiagnosis(channel);
console.log(diagnosis.quadrant); // 'HELPFUL' | 'MARGINAL' | 'HARMFUL'

Install

npm install @polyweave/protocol-metrics

Description

ProtocolMetrics tracks correction/corruption counts for LLM protocol steps and produces channel estimates using Jeffreys smoothing. Each step is modeled as a 2x2 confusion matrix: a protocol can turn wrong answers right (correction, c) or right answers wrong (corruption, gamma).

Key concepts:

  • Channel: a 2x2 correction/corruption kernel estimated from observed outcomes
  • Slice: a difficulty or domain partition for per-stratum gating decisions
  • Phase plane: classifies each channel as HELPFUL, MARGINAL, or HARMFUL
  • Mixture shift: detects bias when calibration and deployment distributions differ

API

createProtocolMetrics(options)

Create a metrics accumulator.

Parameters:

  • options.defaultSlice: default slice name when event has none (default 'all')

Returns an object with record, estimate, getCounts, listEstimates methods.

estimateChannel(counts, options)

Compute correction/corruption channel from observed counts.

Parameters:

  • counts: { n00, n01, n10, n11 } where n01 = corrected, n10 = corrupted
  • options.smoothing: Jeffreys smoothing prior (default 0.5)

Returns { protocolId, slice, counts, p0, p1, c, gamma, delta, predictedDelta, breakEvenCorrection }.

predictPostAccuracy(p0, channel)

Predict accuracy after applying a channel to a population with base accuracy p0.

shouldActivateStep(p0, channel, options)

Gate decision: activate or suppress based on expected gain.

compositionGap(t02Direct, t01, t12)

Compare direct kernel to composed (sequential) kernel. Returns mean and max gap.

estimateSliceChannels(events, options)

Estimate per-slice channels from event list.

predictTransfer(calibChannel, targetP0)

Predict accuracy on a target population from calibration channel.

transferResidual(calibChannel, targetCounts)

Compute signed prediction error for a transfer prediction.

mixtureShiftBias(sliceChannels, deploymentWeights, calibrationWeights)

Detect pooled vs slice-conditioned bias from distribution shift.

sliceGatingDecisions(sliceChannels, options)

Return per-slice activation decisions.

phasePlaneDiagnosis(channel, options)

Classify channel into HELPFUL / MARGINAL / HARMFUL quadrant.

Tests

npm test

Tests cover correction counts, channel estimation, composition gap, slice channels, transfer residuals, mixture shift bias, gating decisions, and phase plane classification.

Requirements

Node.js 18 or later. ESM only.

License

MIT

Dependencies

Development Dependencies

ID Version
@rigor/core *

Keywords

polyweave metrics protocol llm correction corruption channel gating
Details
npm
2026-07-11 00:55:55 +00:00
7
John Dvorak
SEE LICENSE IN LICENSE
4.4 KiB
Assets (1)
protocol-metrics-1.0.2.tgz 4.4 KiB
Versions (4) View all
1.0.5 2026-07-11
1.0.4 2026-07-11
1.0.2 2026-07-11
1.0.0 2026-07-10