@polyweave/optimization (1.0.6)
Installation
@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/npm install @polyweave/optimization@1.0.6"@polyweave/optimization": "1.0.6"About this package
@polyweave/optimization
Performance and cost optimization wrappers for Polyweave adapter pipelines. Provides transparent request batching (coalescing START…END sequences into BATCH frames) and distribution-based cost correction with online learning from RECEIPT frames.
Synopsis
import { createAutoBatcher, createCostCorrectionWrapper } from '@polyweave/optimization';
const autoBatcher = createAutoBatcher(adapter, { maxBatchSize: 10, maxWaitMs: 50 });
const costCorrector = createCostCorrectionWrapper(autoBatcher, {
outputDistribution: { type: 'lognormal', mean: 150, stddev: 80 }
});
Install
npm install @polyweave/optimization
Why
LLM adapter pipelines benefit from two transparent optimizations. First, request coalescing: many small START…END sequences can be batched into a single BATCH frame for adapters that support batch processing, reducing HTTP overhead and enabling provider-side batch pricing. Second, cost correction: naive cost estimates based on max_tokens are systematically pessimistic. By injecting a distribution model (normal, lognormal, empirical, or ratio-based) into the response pipeline, COST frames are corrected with learned percentiles from RECEIPT frames, tightening budget accuracy over time.
API
createAutoBatcher(innerDuplex [, options])
Wraps an inner duplex adapter, intercepting START/END-bounded request sequences and coalescing them into BATCH frames. Uses two triggers: count-based (when maxBatchSize requests are pending) and time-based (when maxWaitMs elapses since the first pending request). BATCH_RESPONSE frames are demuxed transparently back to individual streams.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
innerDuplex |
IFrameDuplex |
Yes | — | Wrapped adapter (must support BATCH frames) |
options.maxBatchSize |
number |
No | 10 |
Max requests per batch (count trigger) |
options.maxWaitMs |
number |
No | 50 |
Max wait time before flushing (time trigger) |
options.enabled |
boolean |
No | true |
If false, returns innerDuplex unchanged (passthrough) |
options.durability.enabled |
boolean |
No | false |
Emit durability META frames on state changes |
options.durability.wrapperId |
string |
No | undefined |
Unique wrapper ID for durability state |
options.name |
string |
No | null |
Name used in topology reports |
Return value: IFrameDuplex — a merged duplex combining the bidirectional wrapper with an observable state stream. Sink is a split sink; source is a join source.
Frame batching behavior:
STARTframes begin collection for a new request;streamIdis stamped if missingTEXT,IMAGE,PARAM,TOOL_CALL,TOOL_RESULT,AUDIO,VIDEO,METAframes within a START…END window are accumulatedENDframes complete a request and move it to the pending batch queueQUITframes flush the pending batch immediatelyBATCH_RESPONSEframes are demuxed: frames are emitted individually with their originalstreamId- Frames outside a START…END window are passed through unchanged
Error conditions: Returns innerDuplex unchanged if enabled is false. Does not throw on invalid input — unknown frames outside a request scope are passed through. BATCH_RESPONSE frames with missing streamId are logged as warnings and skipped.
createCostCorrectionWrapper(innerDuplex [, options])
Wraps an inner duplex, intercepting COST frames on the response path and applying distribution-based corrections. Optionally learns from RECEIPT frames to improve future estimates via online statistics (Welford's algorithm).
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
innerDuplex |
IFrameDuplex |
Yes | — | Wrapped adapter |
options.outputDistribution |
Object |
No | null |
Distribution for output token correction |
options.outputDistribution.type |
string |
No | — | 'normal' | 'lognormal' | 'empirical' | 'ratio' |
options.outputDistribution.mean |
number |
No | — | Mean output tokens (for 'normal' / 'lognormal') |
options.outputDistribution.stddev |
number |
No | — | Standard deviation (for 'normal' / 'lognormal') |
options.outputDistribution.percentiles |
Object |
No | — | Direct percentiles { p5, p50, p95, ... } (for 'empirical') |
options.outputDistribution.typicalRatio |
number |
No | 0.4 |
Typical / max_tokens ratio (for 'ratio') |
options.outputDistribution.maxRatio |
number |
No | 0.9 |
Max expected / max_tokens ratio (for 'ratio') |
options.inputCorrection |
Object |
No | null |
Correction for input token estimates |
options.inputCorrection.multiplier |
number |
No | — | Multiply input estimate by this |
options.inputCorrection.additive |
number |
No | — | Add this to input estimate |
options.learning |
Object |
No | { enabled: false } |
Learning configuration |
options.learning.enabled |
boolean |
No | false |
Enable learning from RECEIPT frames |
options.learning.minSamples |
number |
No | 10 |
Minimum samples before using learned stats |
options.learning.blendFactor |
number |
No | 0.5 |
Blend factor for learned vs provided (0–1) |
options.onStatsUpdate |
function |
No | null |
Callback (stats) => void when stats updated |
options.name |
string |
No | null |
Name used in topology reports |
Return value: IFrameDuplex — bidirectional wrapper. Passes all request frames through unchanged. Intercepts COST frames on response to correct estimates. Intercepts RECEIPT frames (with content.final === true) for learning.
Corrected COST frame additions: The output frame includes originalEstimates (pre-correction), correction metadata (distribution source, parameters, percentiles, learned sample count), recalculated estimates (input/output tokens and costs), corrected bounds (p5, p25, p50, p75, p95, max with costs), and a forwardEstimate ({ min, typical, max } output tokens).
Distribution blending: When learning is enabled and minSamples is reached, learned percentiles are blended with provided percentiles using blendFactor. Source is annotated as 'provided', 'learned', or 'blended'.
Error conditions: If a COST frame's provider/model has no cost config in @polyweave/costs, the frame is passed through unchanged. Does not throw — all errors are handled silently within correction logic.
createCostCorrectionWrapperWithStats(innerDuplex [, options])
Extended variant of createCostCorrectionWrapper that exposes live statistics access. Same options signature.
Return value: { duplex: IFrameDuplex, getStats: () => stats, setDistribution: (newDistribution) => void, getDistribution: () => currentDistribution }
getStats() returns { inputStats: { count, mean, stddev, percentiles }, outputStats: { count, mean, stddev, percentiles }, costRatioStats: { count, mean, stddev, percentiles } }.
normalPercentiles(mean, stddev)
Calculates percentile table from a normal distribution. Z-scores: p5 (−1.645), p10, p25, p50 (0), p75, p90, p95 (1.645), p99 (2.326). Returns { p5: number, ..., p99: number } with values clamped to ≥1.
lognormalPercentiles(mean, stddev)
Calculates percentile table from a lognormal distribution using mean/stddev of the actual (non-log) distribution. Same return shape as normalPercentiles.
inferDistributionFromPercentiles(percentiles)
Infers distribution parameters from provided percentile values. Requires at least p50. Uses skew ratio (p95−p50)/(p50−p5) to choose normal vs lognormal. Returns { mean, stddev, type } with 'normal' or 'lognormal'.
createOnlineStats()
Returns an online statistics accumulator using Welford's algorithm. API: push(value), count (getter), mean (getter), variance (getter), stddev (getter), percentile(p: 0–100), getPercentiles() → { p5, p10, p25, p50, p75, p90, p95, p99 }, toJSON() → { count, mean, stddev, percentiles }. Keeps at most 1000 observations for percentile calculation.
Tests
npm test
Tests cover: auto-batcher creation and request coalescing, START/END accumulation, batch flush triggers (count and time), BATCH_RESPONSE demuxing, passthrough mode, cost correction with normal/lognormal/ratio/empirical distributions, learning from RECEIPT frames, blended estimates, input correction, online statistics (Welford algorithm convergence), and js-rigor property tests.
Requirements
- Node.js >= 22.0.0
- ESM (
"type": "module") @polyweave/core(peer)@polyweave/costs(peer)
Dependencies
Development Dependencies
| ID | Version |
|---|---|
| @rigor/core | * |
Peer Dependencies
| ID | Version |
|---|---|
| @polyweave/core | * |
| @polyweave/costs | * |