@polyweave/protocol-metrics (1.0.5)
Installation
@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/npm install @polyweave/protocol-metrics@1.0.5"@polyweave/protocol-metrics": "1.0.5"About this package
@polyweave/protocol-metrics
Correction/corruption telemetry and gating primitives for LLM protocol steps.
Published as @polyweave/protocol-metrics.
Synopsis
import {
createProtocolMetrics,
estimateChannel,
phasePlaneDiagnosis,
sliceGatingDecisions,
compositionGap,
mixtureShiftBias,
transferResidual
} from '@polyweave/protocol-metrics';
const metrics = createProtocolMetrics();
metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: false, afterCorrect: true });
metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: false, afterCorrect: true });
metrics.record({ protocolId: 'qa-step', slice: 'easy', beforeCorrect: true, afterCorrect: true });
metrics.record({ protocolId: 'qa-step', slice: 'hard', beforeCorrect: true, afterCorrect: false });
const channel = metrics.estimate('qa-step', 'easy');
console.log(channel.c); // correction rate (wrong → right), e.g. ~0.833
console.log(channel.gamma); // corruption rate (right → wrong), e.g. ~0.167
console.log(channel.delta); // net accuracy change
const diagnosis = phasePlaneDiagnosis(channel);
console.log(diagnosis.quadrant); // 'HELPFUL' | 'MARGINAL' | 'HARMFUL'
console.log(diagnosis.expectedGain);
const decisions = sliceGatingDecisions({ easy: channel, hard: metrics.estimate('qa-step', 'hard') });
console.log(decisions.map(d => `${d.slice}: ${d.activate}`));
Install
npm install @polyweave/protocol-metrics
Why
When you chain LLM protocol steps (rewrite → verify → refine), each step can correct prior mistakes or introduce new ones. This package models each step as a 2×2 confusion matrix (a "channel") tracking correction and corruption rates. It provides Jeffreys-smoothed estimation, per-slice (stratum) breakdowns, phase-plane classification (HELPFUL/MARGINAL/HARMFUL), gating decisions, composition gap analysis (direct vs. composed kernel fidelity), transfer prediction (calibration → deployment population shift), and mixture shift bias detection. All functions are pure — no side effects, no dependencies.
API
createProtocolMetrics(options?)
Creates a metrics accumulator that buckets events by (protocolId, slice) and produces channel estimates.
Signature
function createProtocolMetrics(options?: {
defaultSlice?: string
}): {
record: (event: RecordEvent) => ChannelEstimate,
estimate: (protocolId?: string, slice?: string) => ChannelEstimate,
getCounts: (protocolId?: string, slice?: string) => Counts,
listEstimates: () => ChannelEstimate[]
}
Parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.defaultSlice |
string |
No | 'all' |
Slice name used when an event has no slice property. |
record(event)
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
event.protocolId |
string |
No | 'default' |
Protocol step identifier. |
event.slice |
string |
No | defaultSlice |
Difficulty/domain partition label. |
event.beforeCorrect |
boolean |
Yes | — | Whether the answer was correct before the protocol step. |
event.afterCorrect |
boolean |
Yes | — | Whether the answer was correct after the protocol step. |
Returns the ChannelEstimate for the updated bucket.
estimate(protocolId?, slice?)
Returns the ChannelEstimate for the given bucket (or a zero-count estimate if no data).
getCounts(protocolId?, slice?)
Returns the raw counts { protocolId, slice, n00, n01, n10, n11 } for the bucket.
listEstimates()
Returns ChannelEstimate[] for all recorded buckets.
estimateChannel(counts, options?)
Computes a correction/corruption channel from observed outcome counts using Jeffreys smoothing.
Signature
function estimateChannel(
counts: { n00: number, n01: number, n10: number, n11: number, protocolId?: string, slice?: string },
options?: { smoothing?: number }
): ChannelEstimate
Parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
counts.n00 |
number |
No | 0 |
Times answer was wrong → wrong. |
counts.n01 |
number |
No | 0 |
Times answer was wrong → right (correction). |
counts.n10 |
number |
No | 0 |
Times answer was right → wrong (corruption). |
counts.n11 |
number |
No | 0 |
Times answer was right → right. |
options.smoothing |
number |
No | 0.5 |
Jeffreys prior added to both correction and corruption counts. |
Return Value — ChannelEstimate
{
protocolId: string,
slice: string,
counts: { n00, n01, n10, n11, n0, n1, total },
p0: number, // pre-step accuracy (n1 / total)
p1: number, // post-step accuracy ((n01 + n11) / total)
c: number, // correction rate = (n01 + smooth) / (n0 + 2*smooth)
gamma: number, // corruption rate = (n10 + smooth) / (n1 + 2*smooth)
delta: number, // raw accuracy delta (p1 - p0)
predictedDelta: number, // (1-p0)*c - p0*gamma
breakEvenCorrection: number // amplification * gamma; Infinity if p0 >= 1
}
predictPostAccuracy(p0, channel)
Predicts post-step accuracy given base accuracy p0 and a channel.
function predictPostAccuracy(p0: number, channel: ChannelEstimate): number
// Returns: p0 + (1 - p0) * channel.c - p0 * channel.gamma
shouldActivateStep(p0, channel, options?)
Gate decision: should this protocol step run for a population with base accuracy p0?
Signature
function shouldActivateStep(
p0: number,
channel: ChannelEstimate,
options?: { costThreshold?: number }
): { activate: boolean, expectedGain: number, threshold: number }
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.costThreshold |
number |
No | 0 |
Minimum expected gain required to activate. |
Returns activate: true when expectedGain > costThreshold.
compositionGap(t02Direct, t01, t12)
Compares a direct 2×2 transition matrix to the composition of two sequential matrices (step 0→1 then 1→2). Measures how much the composed kernel diverges from the directly observed kernel.
Signature
function compositionGap(
t02Direct: [[number, number], [number, number]],
t01: [[number, number], [number, number]],
t12: [[number, number], [number, number]]
): { mean: number, max: number, composed: [[number, number], [number, number]] }
Each matrix is a row-stochastic 2×2: [[p(0→0), p(0→1)], [p(1→0), p(1→1)]]. The composed matrix is t01 × t12 via standard matrix multiplication.
estimateSliceChannels(events, options?)
Estimates per-slice channels from an array of outcome events.
Signature
function estimateSliceChannels(
events: Array<{ beforeCorrect: boolean, afterCorrect: boolean, slice?: string }>,
options?: { sliceFn?: (event) => string, protocolId?: string, smoothing?: number }
): { [slice: string]: ChannelEstimate }
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.sliceFn |
function |
No | (e) => e.slice || 'all' |
Custom slice key extractor. |
options.protocolId |
string |
No | 'step' |
Protocol identifier applied to all estimates. |
options.smoothing |
number |
No | 0.5 |
Jeffreys smoothing prior. |
predictTransfer(calibChannel, targetP0)
Predicts accuracy on a target population using a calibration channel.
function predictTransfer(calibChannel: ChannelEstimate, targetP0: number): number
// Delegates to predictPostAccuracy(targetP0, calibChannel)
transferResidual(calibChannel, targetCounts)
Computes the signed prediction error between predicted and actual accuracy for a target population.
Signature
function transferResidual(
calibChannel: ChannelEstimate,
targetCounts: { n00: number, n01: number, n10: number, n11: number }
): TransferResidual
Return Value
{
predicted: number, // predictTransfer(calibChannel, targetP0)
actual: number, // targetP1
residual: number, // actual - predicted
standardizedResidual: number, // residual / SE, where SE = sqrt(predicted * (1-predicted) / n)
targetN: number, // total target observations
targetP0: number, // target pre-step accuracy
targetP1: number, // target post-step accuracy
calibC: number, // calib channel correction rate
calibGamma: number // calib channel corruption rate
}
mixtureShiftBias(sliceChannels, deploymentWeights, calibrationWeights)
Detects bias when pooled estimation (weighted by observation count) differs from slice-conditioned estimation (weighted by deployment distribution). This reveals Simpson's paradox in channel estimation.
Signature
function mixtureShiftBias(
sliceChannels: { [slice: string]: ChannelEstimate },
deploymentWeights?: { [slice: string]: number },
calibrationWeights?: { [slice: string]: number }
): MixtureShiftBias
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
sliceChannels |
object |
Yes | — | Map of slice name to ChannelEstimate. |
deploymentWeights |
object |
No | uniform | Per-slice weight in the deployment distribution. |
calibrationWeights |
object |
No | uniform | Per-slice weight in the calibration distribution. |
Return Value
{
pooledC: number, // observation-count-weighted c
pooledGamma: number, // observation-count-weighted gamma
sliceConditionedC: number, // deployment-weight-weighted c
sliceConditionedGamma: number, // deployment-weight-weighted gamma
biasC: number, // pooledC - sliceConditionedC
biasGamma: number, // pooledGamma - sliceConditionedGamma
sliceKeys: string[] // sorted slice names
}
sliceGatingDecisions(sliceChannels, options?)
Returns per-slice activation decisions by applying shouldActivateStep to each slice.
Signature
function sliceGatingDecisions(
sliceChannels: { [slice: string]: ChannelEstimate },
options?: { costThreshold?: number }
): GatingDecision[]
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.costThreshold |
number |
No | 0 |
Minimum expected gain threshold. |
Return Value — GatingDecision[]
Each element:
{
slice: string,
p0: number,
p1: number,
c: number,
gamma: number,
expectedGain: number,
activate: boolean,
counts: { n00, n01, n10, n11, n0, n1, total }
}
Sorted by slice name ascending.
phasePlaneDiagnosis(channel, options?)
Classifies a channel into the HELPFUL / MARGINAL / HARMFUL quadrant based on correction surplus.
Signature
function phasePlaneDiagnosis(
channel: ChannelEstimate,
options?: { p0?: number }
): PhasePlaneDiagnosis
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.p0 |
number |
No | channel.p0 (or 0.5 if not finite) |
Base accuracy for computing surplus. |
Return Value
{
c: number,
gamma: number,
p0: number,
amplification: number, // p0 / (1-p0); Infinity if p0 >= 1
breakEvenC: number, // amplification * gamma
surplus: number, // c - breakEvenC
expectedGain: number, // surplus * (1 - p0)
quadrant: string // 'HELPFUL' | 'MARGINAL' | 'HARMFUL'
}
Quadrant classification:
'HELPFUL'—surplus > 0'MARGINAL'—surplus > -0.05(near break-even)'HARMFUL'—surplus <= -0.05
Error Conditions
All 12 exported functions are pure and throw no errors. Parameters have safe defaults (0 for missing counts, 0.5 for smoothing). Division by zero is guarded by checking total > 0, n > 0, etc. Returns Infinity for breakEvenCorrection and amplification when p0 >= 1.
Tests
npm test
Covers:
- protocol-metrics.test.js — Correction count accumulation, channel estimation with Jeffreys smoothing, composition gap (direct vs composed 2×2), slice channel estimation, transfer residual computation, mixture shift bias detection, slice gating decisions, phase plane classification (all three quadrants and edge cases).
- rigor.test.js — Property-based tests: monotonicity of correction/corruption with more data, compose-and-compare invariants, gating decision consistency, phase plane boundary conditions.
Requirements
- Node.js 22 or later.
- ESM only (
"type": "module"). - Zero runtime dependencies.
Dependencies
Development Dependencies
| ID | Version |
|---|---|
| @rigor/core | * |