@polyweave/uncertainty (1.0.7)
Installation
@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/npm install @polyweave/uncertainty@1.0.7"@polyweave/uncertainty": "1.0.7"About this package
@polyweave/uncertainty
Uncertainty metrics for LLM outputs: radial dispersion, semantic entropy, ensemble consistency, and hallucination detection.
Synopsis
import { createUncertaintyWrapper, FrameUncertaintyMetric } from '@polyweave/uncertainty';
const wrapped = createUncertaintyWrapper(llmAdapter, {
config: { sampleCount: 5, metric: FrameUncertaintyMetric.RADIAL_DISPERSION }
});
wrapped.sink.write({ type: 'TEXT', content: { text: 'What is the capital of France?' } });
wrapped.sink.write({ type: 'END' });
wrapped.source.pipe(outputSink); // receives enriched TEXT + META(uncertainty) + END
import { radialDispersionScore, semanticEntropy, ensembleConsistency } from '@polyweave/uncertainty';
const samples = [
{ text: 'Paris is the capital of France.', embedding: [0.1, 0.2, 0.3] },
{ text: 'The capital of France is Paris.', embedding: [0.12, 0.19, 0.31] },
{ text: 'France has Paris as its capital.', embedding: [0.11, 0.21, 0.29] }
];
const dispersion = radialDispersionScore(samples); // sum of L1 distances from centroid
const entropy = semanticEntropy(samples); // clustering-based entropy
const consistency = ensembleConsistency(samples); // weighted pairwise similarity
Install
npm install @polyweave/uncertainty
Why
LLM outputs are stochastic. Running the same prompt multiple times produces different responses, and without measurement you can't tell whether the model is confident or guessing. This package provides a frame-stream wrapper (createUncertaintyWrapper) that fans out parallel samples, optionally computes embeddings, and enriches responses with uncertainty metadata — plus a comprehensive library of standalone metrics (logprob analysis, semantic entropy, SESE structural entropy, HHEM hallucination detection, prompt perturbation sensitivity, multi-turn confidence, citation offset resolution) for programmatic use.
API
FrameUncertaintyMetric
Enum of available metrics:
| Value | String | Description |
|---|---|---|
RADIAL_DISPERSION |
'radial_dispersion' |
L1 distance from centroid in embedding space |
WEIGHTED_RADIAL_DISPERSION |
'weighted_radial_dispersion' |
Same but weighted by response probability |
SEMANTIC_ENTROPY |
'semantic_entropy' |
Cluster responses and compute entropy |
ENSEMBLE_CONSISTENCY |
'ensemble_consistency' |
Measure agreement between samples |
TOKEN_ENTROPY |
'token_entropy' |
Average per-token entropy from logprobs |
HYBRID |
'hybrid' |
60% radial dispersion + 40% ensemble consistency |
createUncertaintyWrapper(adapter, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
adapter |
object |
yes | — | LLM adapter with sink/source |
options |
object |
no | {} |
Configuration |
options.embeddingAdapter |
object |
no | null |
Embedding adapter for true radial dispersion |
options.config |
object |
no | validated defaults | Uncertainty estimation config |
Returns { source, sink } — a frame duplex.
Request path (input → adapter): Buffers START/PARAM/TEXT/END, then on END fans out N parallel sample requests to the inner adapter. PARAM frames with keys uncertaintySamples, uncertaintyThreshold, uncertaintyMetric, temperature, embeddingModel override the config at runtime.
Response path (adapter → output): Collects sample responses (by streamId or sequential fallback), optionally computes embeddings, then emits either the best sample or all samples (if emitAllSamples is true) followed by a META frame with uncertainty metadata.
Frame protocol:
| Inbound | Outbound |
|---|---|
| START → PARAM(s) → TEXT → END | (buffered, fans out N samples) |
| (after fan-out) ← adapter | TEXT (each collected sample), META (uncertainty envelope), END |
| — | ERROR (with code: UNCERTAINTY_ERROR, INSUFFICIENT_SAMPLES, ALL_SAMPLES_FAILED, NO_VALID_SAMPLES) |
Config validation (validateUncertaintyConfig):
| Key | Type | Default | Range | Description |
|---|---|---|---|---|
sampleCount |
number |
5 |
2–20 | Number of parallel samples |
metric |
string |
'radial_dispersion' |
any FrameUncertaintyMetric value |
Uncertainty metric |
threshold |
number |
0.5 |
0–1 | Uncertainty threshold |
weighted |
boolean |
false |
— | Use weighted radial dispersion |
temperature |
number |
0.7 |
0–2 | Sampling temperature |
normalizeEmbeddings |
boolean |
true |
— | Normalize embeddings before dispersion |
semanticThreshold |
number |
0.8 |
0–1 | Similarity threshold for semantic clustering |
emitAllSamples |
boolean |
false |
— | Emit all samples instead of just the best |
includePerSampleScores |
boolean |
true |
— | Include per-sample scores in output |
samplingTimeout |
number |
30000 |
>= 1000 | Sampling timeout (ms) |
maxRetries |
number |
2 |
0–5 | Max retries for failed samples |
failOnPartial |
boolean |
false |
— | Fail if minSuccessfulSamples not met |
minSuccessfulSamples |
number |
3 |
>= 1 | Minimum required successful samples |
Error conditions (emitted as ERROR frames, not thrown):
UNCERTAINTY_ERROR— when text content is empty after extracting from buffered TEXT framesINSUFFICIENT_SAMPLES— whensuccessfulCount < config.minSuccessfulSamplesandfailOnPartialis trueALL_SAMPLES_FAILED— when zero samples succeededNO_VALID_SAMPLES— when all collected samples have empty text
Radial Dispersion (src/radialDispersion.js)
radialDispersionScore(samples, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
samples |
Array<{ embedding }> |
yes | — | Samples with embedding array (or vector, embeddingVector, values) |
options |
object |
no | {} |
Options |
options.normalizeEmbeddings |
boolean |
no | true |
L2-normalize embeddings before computation |
options.embedding / options.getEmbedding |
function |
no | — | Custom embedding extractor: sample => number[] |
Returns the sum of L1 distances from each sample's embedding to the centroid. Returns null if dimension validation fails or no samples have embeddings.
weightedRadialDispersionScore(samples, options)
Same as radialDispersionScore but weights each sample's contribution by weightFromResponse(sample.sample, options) (uses weight, score, prob, probability, logprob, or defaults to 1).
radialDispersionPerSample(samples, options)
Returns Array<{ index, sample, score }> — L1 distance from each sample to the centroid. Returns [] if validation fails.
weightedRadialDispersionPerSample(samples, options)
Same but weighted.
radialDispersionCentroid(samples, options)
Returns the centroid vector (array of numbers). Returns null if validation fails.
weightedRadialDispersionCentroid(samples, options)
Same but weighted.
rds, rdsw
Short aliases for radialDispersionScore and weightedRadialDispersionScore.
No functions in this module throw.
Semantic Metrics (src/semantic.js)
clusterResponses(samples, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
samples |
array |
yes | — | Sample objects with text and optionally embedding |
options.threshold / options.similarityThreshold |
number |
no | 0.8 |
Similarity threshold for cluster assignment |
options.similarity / options.similarityFn |
function |
no | pickSimilarity |
Custom similarity function |
Returns an array of clusters: { representative, members, weight }. Uses greedy assignment: each sample joins the first cluster whose representative similarity >= threshold.
semanticEntropy(samples, options)
Returns entropy over response clusters. null if no clusters or zero total weight.
semanticConsistency(samples, options)
Returns weighted pairwise similarity (0–1). null if fewer than 2 items or zero total weight.
semanticUncertainty(samples, options)
Returns 1 - semanticConsistency(samples, options). null if consistency is not finite.
lexicalSimilarityScore(samples, options)
Returns average pairwise Jaccard similarity (via pickSimilarity). null if fewer than 2 items.
logprobWeightedSimilarity(samples, options)
Weighted pairwise similarity where weights come from weightFromResponse. null if zero total weight.
nliConsistencyScore(nliResult)
Given an NLI result with entailment/logitEntailment and contradiction/logitContradiction fields, returns contradiction - entailment. null if either is not finite.
ensembleNliScore(nliResults)
Returns the minimum nliConsistencyScore across an array of NLI results. null if no valid scores.
selfCheckNli(samples, options)
Returns average (or min/max if options.mode is 'min' or 'max') of contradiction/contradictionProb/contradictionProbability/probability/prob values from samples. null if no valid scores.
logprobFromResponse(response)
Extracts logprob from response.logprob / response.logProbability / response.averageLogprob / response.avgLogprob / average of response.logprobs array. Returns null if none found.
expressionLogprobRatio(referenceResponse, alternativeResponse)
Returns logprobFromResponse(alternative) - logprobFromResponse(reference). null if either is not finite.
uncertaintyUnderExpression(referenceResponse, alternativeResponses, options)
Returns the average (or min/max) of expressionLogprobRatio(reference, alt) across alternatives. null if no valid ratios.
logprobWeightFromResponse(response, options)
Returns the probability weight from a response: prob → probability → logprobToProb(logprobFromResponse(...)) → weightFromResponse(...). Always >= 0.
No functions in this module throw.
Ensemble (src/ensemble.js)
ensembleConsistency(samples, options)
Weighted average pairwise similarity. Same implementation as logprobWeightedSimilarity. null if fewer than 2 items.
ensembleUncertainty(samples, options)
Returns 1 - clampNumber(ensembleConsistency(samples, options), 0, 1). null if consistency is not finite.
ensembleConsensus(samples, options)
Returns { response, score } — the sample with the highest weighted average similarity to all other samples.
No functions in this module throw.
Logprob Metrics (src/logprobMetrics.js)
averageLogProbability(logprobs)
Average logprob across normalized tokens. null if no tokens.
averageProbability(logprobs)
Average Math.exp(logprob) across tokens. null if no tokens.
minimumTokenLogprob(logprobs)
Minimum logprob across tokens. null if no tokens.
minimumTokenProbability(logprobs)
Math.exp(minimumTokenLogprob(logprobs)). null if no tokens.
minKPercentProbability(logprobs, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
options.percent / options.k / options.kPercent |
number |
no | 0.2 |
Fraction of lowest-probability tokens to consider |
options.mode |
string |
no | 'average' |
'average' or 'min' |
Sorts token probabilities ascending, takes the bottom percent fraction, returns average (or min). null if no valid tokens or percent is 0.
normalizeLogprobStats(logprobs, options)
Returns { averageLogprob, averageProb, minimumLogprob, minimumProb, minKPercentProb }.
clampLogprob(value, options)
Clamps to [options.min = -100, options.max = 0]. Returns clampNumber(value, min, max).
lengthNormalizedLogprob(logprobs)
Normalizes logprob by token count. Accepts: a finite number, an object with averageLogprob/avgLogprob/meanLogprob, an object with logprob + tokenCount, an object with logprobSum + tokenCount, or a token array (delegates to averageLogProbability).
logprobRatio(reference, alternative)
Returns lengthNormalizedLogprob(alternative) - lengthNormalizedLogprob(reference). null if either is not finite.
No functions in this module throw.
Entropy (src/entropy.js)
entropyFromProbs(probs, options)
Shannon entropy from an array of probabilities. null if no valid probs or zero total.
entropyFromLogprobs(logprobs, options)
Converts logprobs to probs, then calls entropyFromProbs.
averagePointwiseEntropy(logprobs, options)
Average entropy from top_logprobs of each token. null if no tokens have top_logprobs.
responseSampleEntropy(samples, options)
Entropy over sample weights (extracted from weight, probability, prob, logprob, or default 1). Optionally normalized by log(n). null if no weights.
entropyGap(referenceEntropy, alternativeEntropy)
Returns alternativeEntropy - referenceEntropy. null if either is not finite.
boundedEntropy(value, options)
Clamps to [options.min = 0, options.max = log(2)].
No functions in this module throw.
Token Signals (src/tokenSignals.js)
tokenEntropyFromLogits(logits)
Softmax → Shannon entropy. null if no logits.
tokenProbabilityFromLogits(logits, tokenIndex)
Softmax probability of a specific token index. null if index out of bounds or no logits.
logitMean(logits)
Arithmetic mean. null if no finite logits.
logitVariance(logits)
Population variance. null if fewer than 2 logits or mean is not finite.
logitL2Norm(logits)
Euclidean norm (sqrt of sum of squares). null if no finite logits.
percentileValue(values, percentile)
Value at the given percentile (0–1). null if no values or invalid percentile.
minKPercentile(values, percentile)
Alias for percentileValue.
minKProbability(values, percentile)
Alias for percentileValue.
minKEntropy(values, percentile)
Alias for percentileValue.
excludeLeadingTokens(tokens, count)
Returns array with the first count tokens removed. Default count is 1.
normalizeTokenIndices(indices, options)
Normalizes indices to 0–1 range: (value - min) / (max - min). Returns [0] if all values are equal.
No functions in this module throw.
SESE — Structural Entropy of Semantic Embeddings (src/sese.js)
adjustDirectedGraph(matrix, options)
Ensures graph is strongly connected using Tarjan's SCC algorithm. Adds epsilon edges from sinks to sources. Row-normalizes the result.
stationaryDistribution(matrix, options)
Power iteration (up to options.iterations = 200, tolerance options.tolerance = 1e-9). Returns the stationary distribution π.
oneDimStructuralEntropy(matrix, options)
Computes 1D structural entropy: -Σ πᵢ log₂(πᵢ) after adjusting the directed graph and computing stationary distribution.
seseEncodingTree(matrix, options)
Builds a hierarchical encoding tree with max height options.maxHeight = 4. Greedily merges leaf nodes that maximize structural entropy reduction.
seseScore(matrix, options)
Computes structural entropy of the encoding tree.
buildEntailmentGraph(samples, options)
Builds an adjacency matrix using options.entailment(sampleA, sampleB) as edge weights (clamped 0–1). Returns a zero matrix if entailment is not a function.
seseFromSamples(samples, options)
Shortcut: builds entailment graph (if options.matrix is not provided), then computes seseScore. null if matrix is empty.
No functions in this module throw.
HHEM — Hierarchical Hallucination Ensemble Metric (src/hhem.js)
hhemScore(response, knowledge, options)
Returns a similarity score between response and knowledge. Uses options.scorer / options.scoreFn if provided, otherwise falls back to Jaccard text similarity (defaultTextSimilarity).
classifyHallucination(score, options)
Returns true if score < options.threshold (default 0.5). Returns null if score is not finite.
segmentText(text, options)
Splits text into segments. Modes: 'sentence' (default, split on [.!?]), 'newline' (split on newlines), 'chunk' (fixed-size chunks of options.chunkSize = 200).
hhemSegmentedScore(response, options)
Segments the response, scores each segment against options.knowledgeProvider(segment) or options.knowledge, returns the minimum score. If any segment is below threshold and a penalty is set (default 0.5), multiplies the minimum score by the penalty.
hhemDecision(response, options)
Returns { score, hallucinated } using hhemScore and classifyHallucination.
hhemSegmentedDecision(response, options)
Returns { score, hallucinated } using hhemSegmentedScore and classifyHallucination.
No functions in this module throw.
Prompt Perturbation (src/promptPerturbation.js)
promptPerturbationDiscrepancy(originalResponse, perturbedResponse, options)
Returns 1 - similarity (default mode) or -ratio (logprob mode) between original and perturbed responses. null if either is falsy.
promptPerturbationScore(originalResponse, perturbedResponses, options)
Average (or min/max if options.aggregate = 'min'/'max') of discrepancies across perturbed responses.
promptPerturbationDecision(originalResponse, perturbedResponses, options)
Returns { score, hallucinated } where hallucinated = score >= threshold (default threshold 0.5).
promptLogprobShift(originalResponse, perturbedResponse)
Returns expressionLogprobRatio(original, perturbed). null if ratio is not finite.
promptLogprobStats(originalResponse, perturbedResponses)
Returns { average, min, max } of logprob ratios across perturbed responses. null if no valid ratios.
promptResponseSummary(response)
Returns { text, logprob } of a response.
No functions in this module throw.
Multi-Turn Confidence (src/multiTurnConfidence.js)
normalizeVerbalConfidence(value)
Divides by 100 and clamps to 0–1. null if value is not a finite number.
buildPTruePrompt(vars, options)
Renders a template with ${question}, ${answer}. Uses options.template or a default prompt asking the model to return "A. True" or "B. False".
buildPSufficientPrompt(vars, options)
Same but asks for "A. Sufficient" or "B. Insufficient".
buildVerbalConfidencePrompt(vars, options)
Same but asks for a confidence number 0–100.
extractBinaryChoiceConfidence(logprobs, options)
Given logprobs with top_logprobs, extracts the normalized probability of options.choice ('A') vs options.other ('B'). Returns prob(A) / (prob(A) + prob(B)) clamped to 0–1. null if either probability is not finite.
selfConsistencyConfidence(samples, answer, options)
Returns matches / total where options.compare(sample, answer) is used for equality. Default comparator is ===.
binaryProbeConfidence(probe, options)
Resolves a binary probe confidence from options.probA → options.probability → probe.probA → probe.probability → Math.exp(probe.logprob) → extractBinaryChoiceConfidence(probe.logprobs). Returns null if nothing resolves.
infoLevel(turnIndex, dialogLength)
Returns turnIndex / dialogLength clamped to 0–1. null if either is not finite or dialogLength <= 0.
buildInfoBins(levels, options)
Returns bin thresholds. If options.equalMass, uses equal-mass binning (sorted values at quantile boundaries). Otherwise returns equal-width thresholds.
infoECE(turns, options)
Expected Calibration Error binned by information level. Each turn must have { level } (or { turnIndex, dialogLength }), { confidence }, and { correct }. Returns error averaged over options.bins = 10 bins. null if no turns have valid levels.
kendallTau(confidences)
Kendall rank correlation coefficient of confidence values. null if fewer than 2 values.
kendallTauAverage(dialogs)
Average of kendallTau across dialogs (each dialog must have confidences array). null if no valid taus.
No functions in this module throw.
Citation Offsets (src/citationOffsets.js)
jaccardTextSimilarity(left, right)
Token-level Jaccard similarity of two strings. Returns 0–1.
resolveCitationOffset(documentText, citation, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
documentText |
string |
yes | — | Full document text |
citation |
object |
yes | — | Citation with quote and startOffset/endOffset (or length) |
options.maxExpand |
number |
no | 120 |
Max characters to expand the search window |
options.step |
number |
no | 8 |
Step size for candidate expansion |
options.minJaccard |
number |
no | 0.35 |
Minimum Jaccard for validity |
First checks exact match at the provided span. Then searches for the quoted text in a window. Then tries candidate spans via maxExpand/step grid and picks the best Jaccard match.
Returns { startOffset, endOffset, quote, matchedText, jaccard, certainty, expanded, valid }.
evaluateCitationOffsets(documentText, citations, options)
Returns { results: Array, validCount, invalidCount, averageCertainty, uncertainty }. Each result is from resolveCitationOffset.
charOverlapMetrics(documentText, goldSpans, predictedSpans)
Returns { precision, recall, f1, intersection, predicted, gold } for character-level span overlap.
No functions in this module throw.
Encyclopedia Integration (src/encyclopediaIntegration.js)
createUncertaintyEnhancedRunner(adapter, options)
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
adapter |
object |
yes | — | LLM adapter |
options.embeddingAdapter |
object |
no | null |
Embedding adapter |
options.sampleCount |
number |
no | 3 |
Parallel samples |
options.threshold |
number |
no | 0.5 |
Uncertainty threshold |
options.metric |
string |
no | FrameUncertaintyMetric.RADIAL_DISPERSION |
Uncertainty metric |
options.normalizeEmbeddings |
boolean |
no | true |
Normalize embeddings |
Returns an object with:
runPrompt({ prompt, systemPrompt, options }, callback)— returns{ text, uncertainty, isUncertain, samples, metadata }wrapper— raw uncertainty wrapperuncertaintyConfig—{ sampleCount, threshold, metric, embeddingEnabled }
createUncertaintySummaryWriter(adapter, options)
Wraps createUncertaintyEnhancedRunner for summary generation. Returns { generate(document, promptTemplate, callback), runner }.
createUncertaintyQuestionAnswerer(adapter, options)
Wraps createUncertaintyEnhancedRunner for QA with abstention. Returns { answer(question, context, answerOptions, callback), runner }.
uncertaintyHgnIntegration
Object with:
UncertaintyAwareLLMActivity— HGN activity wrapping LLM calls with uncertaintyLowUncertaintyPredicate— checks!result.isUncertainUsedEmbeddingPredicate— checksresult.metadata?.usedEmbedding === true
Self-Improving Integration (src/selfImprovingIntegration.js)
wrapRolloutWithUncertainty(rolloutFn, options)
Wraps a rollout function (adapter or plain function) with uncertainty estimation. Returns an async function (input, callback) => Promise<{ text, uncertainty, isUncertain, metadata }>.
wrapJudgeWithUncertainty(judgeFn, options)
Wraps a judge function with abstention support. Runs the judge sampleCount times and checks consistency. Returns { ...representativeJudgment, abstained, uncertainty, consistency, judgments, metadata }.
createUncertaintyObjective(qualityScorer, options)
Creates an objective function returning { quality, rawQuality, uncertainty, isUncertain, objectives: { quality, uncertainty } }. Optionally penalizes high uncertainty: quality * (1 - uncertaintyWeight * uncertainty).
algorithmIntegrations
Object with .gepa, .ace, .grpo — each has a createConfig(baseConfig, uncertaintyOptions) that wraps the config's rollout, scorer, filter, etc.
Utility Functions (src/utils.js)
ensureArray(value)
Returns value if it's an array, otherwise [].
clampNumber(value, min, max)
Clamps value to [min, max]. NaN → min.
normalizePercent(value)
Converts a percent to 0–1. >1 → divide by 100. <=1 → use as-is. null if not finite.
logprobToProb(logprob)
Returns Math.exp(logprob). null if not finite.
weightFromResponse(response, options)
Extracts weight: weight → score → prob → probability → logprobToProb(logprob) → options.defaultWeight → 1.
normalizeLogprobTokens(logprobs)
Normalizes various logprob formats (array of {token,logprob,top_logprobs}, {content,token_logprobs,top_logprobs,tokens}, {logprobs}) to [{ token, logprob, top_logprobs }].
cosineSimilarity(a, b)
Cosine similarity of two arrays. null if lengths don't match, any element is non-finite, or zero norms.
defaultTextSimilarity(a, b)
Jaccard similarity of two strings (case-insensitive, whitespace tokenized).
pickSimilarity(responseA, responseB, options)
Uses options.similarity if provided, otherwise tries cosineSimilarity(resA.embedding, resB.embedding), otherwise falls back to defaultTextSimilarity(resA.text, resB.text).
No functions in this module throw.
Tests
npm test
Tests cover all metrics with expected values, the uncertainty wrapper duplex (request buffering, fan-out, sample collection, embedding phase, result emission), PARAM configuration, END-triggered sampling, emit-all mode, error handling (empty text, insufficient samples, all failed), encyclopedia integration, and self-improving integration.
Requirements
- Node.js >= 22.0.0
- ESM only
@polyweave/core(peer)
Dependencies
Development Dependencies
| ID | Version |
|---|---|
| @polyweave/core | * |
| @rigor/core | * |
Peer Dependencies
| ID | Version |
|---|---|
| @polyweave/core | * |
| @polyweave/errors | * |