Compare commits

...

3 Commits

Author SHA1 Message Date
John Dvorak bf7376b5ad docs: add scenario examples, fix sampling docs, improve pedagogical completeness
- Add actual scenario definition example to qualify.md
- Add stateful test API example to qualify.md
- Fix observe.md sampling section to show explicit config and rates
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:35:38 -07:00
John Dvorak 8d7382417d docs: add paper citations, fix pedagogical issues, improve SKILL.md
- Cite arxiv 2602.23922 (Invariant-Driven Automated Testing) in all major docs
- Add Progressive Complexity section to SKILL.md for LLM guidance
- Fix SKILL.md Fast Start example to use deterministic ID generation
- Fix getting-started.md failure output inconsistency
- Fix auth-patterns.md TypeScript syntax in JS doc
- Fix fastify-structure.md Date.now() in test helper
- Fix observe.md misleading workspace heading
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:34:00 -07:00
John Dvorak 6c39bd0a6c docs: final cleanup and accuracy pass before public push
- Fix const inference bug: wrap inferred contracts with status-code guards
- Add integration test for status-guarded contract inference
- Tighten and deduplicate docs across verify, qualify, getting-started, cli
- Fix broken cross-references and TypeScript→JavaScript conversions
- Fix factual errors: license, Date.now(), sampling defaults, cache env
- Add missing features: --workspace, --generation-profile, json-summary formats
- Move stale extension docs (AUTH-RATE-LIMIT-REVISED, HTTP-EXTENSIONS) to attic
- Update PLUGIN_CONTRACTS_SPEC status to Implemented
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:25:30 -07:00
22 changed files with 2564 additions and 269 deletions
+15 -3
View File
@@ -4,6 +4,8 @@ Behavioral confidence for Fastify services.
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
Supported Node.js versions: 20.x and 22.x.
```bash
@@ -12,6 +14,8 @@ apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users"
```
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
## Cross-Route Failure Example
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
@@ -19,6 +23,8 @@ Add one behavioral contract next to a route schema. APOPHIS can verify cross-rou
**Route:**
```javascript
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
@@ -29,7 +35,7 @@ app.post('/users', {
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${Date.now()}`;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
@@ -47,7 +53,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200
Observed
GET /users/usr-123 returned 404
GET /users/usr-7d865e returned 404
Why this matters
The resource created by POST /users is not retrievable.
@@ -80,6 +86,9 @@ apophis init --preset safe-ci
# 3. Verify
apophis verify --profile quick --routes "POST /users"
# 4. Doctor
apophis doctor
```
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
@@ -87,9 +96,12 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
## Trust and Safety
- **Deterministic replay**: Every failure includes a seed and a one-command replay.
- **Generation profile aliases**: Control test budget with `--generation-profile quick|standard|deep`.
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
## LLM-Safe
@@ -116,4 +128,4 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## License
ISC
MIT
+30 -1
View File
@@ -7,6 +7,8 @@ description: Use this skill when adding or improving APOPHIS contract-driven tes
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
## When To Use
Use this skill when the operator asks to:
@@ -76,6 +78,7 @@ When entering a Fastify codebase:
import Fastify from 'fastify'
import swagger from '@fastify/swagger'
import apophis from 'apophis-fastify'
import crypto from 'crypto'
const app = Fastify()
await app.register(swagger)
@@ -114,8 +117,9 @@ app.post('/users', {
}
}
}, async (req, reply) => {
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
reply.status(201)
return { id: 'usr-1', ...req.body }
return { id, ...req.body }
})
await app.ready()
@@ -350,6 +354,31 @@ Operator framing:
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
## Progressive Complexity
Start simple and add depth only where it pays off:
**Level 1 — Status and shape**: Every route gets an expected status code and key field existence.
```apostl
status:201
response_body(this).id != null
```
**Level 2 — Cross-route behavior**: Constructors check retrievability; mutators check persistence.
```apostl
response_code(GET /users/{response_body(this).id}) == 200
response_body(GET /users/{response_body(this).id}).email == request_body(this).email
```
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
```apostl
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
```
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
Add level 2 before level 4. Do not skip level 2 for resource APIs.
## Anti-Patterns
Do not:
+1 -1
View File
@@ -1,6 +1,6 @@
# APOPHIS Plugin Contract System Specification
## Status: Active design; target version to be assigned
## Status: Implemented
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
+1 -1
View File
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## Operator Resources
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout
- [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
## CTAs
+1 -1
View File
@@ -144,7 +144,7 @@ See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
`getToken` runs per request. Handle refresh inline:
```javascript
let cachedToken: string | null = null
let cachedToken = null
const auth = createAuthExtension({
name: 'jwt-with-refresh',
+4 -4
View File
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
| Environment | Cache | Reason |
|-------------|-------|--------|
| `production` | Disabled | No file I/O, no cache hits needed |
| `test` | Disabled | Tests should be deterministic, no cache pollution |
| `development` | Enabled | Speeds up iterative testing |
| default | Enabled | Backward compatible |
| `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `development` | Enabled by default | Speeds up iterative testing |
| default | Enabled by default | Backward compatible |
## Cache Invalidation
+21 -19
View File
@@ -2,9 +2,11 @@
Inject controlled failures into contract tests to validate resilience guarantees.
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
## Usage
```typescript
```javascript
const result = await fastify.apophis.contract({
depth: 'standard',
chaos: {
@@ -14,7 +16,7 @@ const result = await fastify.apophis.contract({
dropout: { probability: 1 },
corruption: { probability: 1 },
},
})
});
```
## Event Types
@@ -52,35 +54,35 @@ Mutates response bodies. Tests parsing robustness:
response_body(this).id != null
```
## Content-Type Aware Corruption
## Corruption Strategies
Built-in strategies for common formats:
Built-in strategies are content-type agnostic:
| Content-Type | Strategy | Effect |
|-------------|----------|--------|
| `application/json` | Truncate or null field | Removes fields or sets random field to null |
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk |
| `text/event-stream` | Event corrupt | Adds malformed SSE line |
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
| `text/plain` | Truncate | Cuts string in half |
| Strategy | Effect |
|----------|--------|
| `truncate` | Cuts response body short |
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
| `field-corrupt` | Replaces a random field value with corrupted data |
Extension strategies can add content-type-specific behavior if needed.
## Custom Corruption via Extensions
```typescript
```javascript
const myExtension = {
name: 'custom-corrupt',
corruptionStrategies: {
'application/vnd.api+json': (data) => ({
...data as object,
...data,
corrupted: true,
}),
'text/*': (data) => `CORRUPTED:${String(data)}`,
},
}
};
await fastify.register(apophis, {
extensions: [myExtension],
})
});
```
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
@@ -90,7 +92,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
```
Error: Chaos mode is only available in test environment.
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
```
## Interpreting Results
@@ -123,7 +125,7 @@ Failed tests include chaos events in diagnostics:
## Example: Testing Retry Logic
```typescript
```javascript
fastify.get('/data', {
schema: {
'x-ensures': [
@@ -131,7 +133,7 @@ fastify.get('/data', {
'redirect_count(this) <= 3',
],
},
}, handler)
}, handler);
// Test
const result = await fastify.apophis.contract({
@@ -139,5 +141,5 @@ const result = await fastify.apophis.contract({
probability: 0.2,
error: { probability: 1, statusCode: 503 },
},
})
});
```
+33 -4
View File
@@ -10,15 +10,17 @@ Every command accepts these flags:
|---|---|---|
| `--config <path>` | Config file path | Auto-detect |
| `--profile <name>` | Profile name from config | First profile |
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived |
| `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
| `--cwd <path>` | Working directory override | `process.cwd()` |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
| `--quiet` | Suppress non-error output | false |
| `--verbose` | Enable verbose logging | false |
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` |
| `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
| `--workspace` | Run supported commands across workspace packages | false |
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
## Commands
### `apophis init`
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
| Flag | Description |
|---|---|
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
| `--force` | Overwrite existing files |
| `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
| `-f, --force` | Overwrite existing files |
| `--noninteractive` | Skip all prompts, require explicit flags |
**Examples:**
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
| `--changed` | Filter to git-modified routes only |
| `--workspace` | Run across all workspace packages |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
**Examples:**
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|---|---|
| `--mode <mode>` | Filter checks to a specific mode |
| `--strict` | Treat warnings as failures |
| `--workspace` | Run across all workspace packages |
**Checks:**
@@ -210,6 +214,31 @@ apophis migrate --dry-run
apophis migrate --write
```
## Common Tasks
### CI workflow with machine output
```bash
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
```
### Monorepo workspace verification
```bash
apophis verify --workspace --profile quick
apophis doctor --workspace
```
### Replay a failure
```bash
apophis replay --artifact reports/apophis/failure-*.json
```
## Gotchas
- `--changed` requires a git repository
- `migrate` defaults to `--dry-run` (safe by default)
- `--workspace` is only supported by `verify` and `doctor` commands
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
## Exit Codes
| Code | Meaning |
+2 -1
View File
@@ -1,5 +1,6 @@
import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify'
import crypto from 'crypto'
const fastify = Fastify()
@@ -40,7 +41,7 @@ fastify.post('/users', {
}
}
}, async (req, reply) => {
const id = `usr-${Date.now()}`
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
const user = { id, email: req.body.email, name: req.body.name }
users.set(id, user)
reply.status(201)
+3 -1
View File
@@ -374,8 +374,10 @@ import { mkdirSync, rmSync } from 'fs'
import { tmpdir } from 'os'
import { join } from 'path'
let testCounter = 0
export function createTestWorkspace() {
const dir = join(tmpdir(), `apophis-test-${Date.now()}`)
const dir = join(tmpdir(), `apophis-test-${++testCounter}`)
mkdirSync(dir, { recursive: true })
return {
+16 -106
View File
@@ -2,6 +2,8 @@
Get from install to your first behavioral bug in 10 minutes.
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
## Prerequisites
- Node.js 20.x or 22.x
@@ -30,6 +32,8 @@ This creates:
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
```javascript
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
@@ -40,27 +44,20 @@ app.post('/users', {
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${Date.now()}`;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
```
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
## Step 4: Run Verify
```bash
apophis verify --profile quick --routes "POST /users"
```
APOPHIS will:
1. Discover routes from your Fastify app
2. Filter to `POST /users`
3. Generate test data from the schema
4. Execute the route
5. Check the behavioral contract
6. Print pass/fail, seed, and replay command
## Example Failure
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
@@ -75,7 +72,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200
Observed
GET /users/usr-123 returned 404
GET /users/usr-7d865e returned 404
Why this matters
The resource created by POST /users is not retrievable.
@@ -100,111 +97,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
## Next Steps
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
- Run all routes: `apophis verify --profile quick`
- Run only changed routes in CI: `apophis verify --profile ci --changed`
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
- Requires a git repository.
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
## Config Reference
```javascript
// apophis.config.js
export default {
mode: 'verify',
profile: 'quick',
profiles: {
quick: {
name: 'quick',
mode: 'verify',
preset: 'safe-ci',
routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
}
},
presets: {
'safe-ci': {
name: 'safe-ci',
depth: 'quick',
timeout: 5000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
}
}
};
```
For the full configuration reference, see [CLI Reference](cli.md).
## Monorepo Workspaces
APOPHIS supports workspace-wide operations with the `--workspace` flag.
### Root package.json scripts
```json
{
"scripts": {
"apophis:verify": "apophis verify --workspace --profile quick",
"apophis:doctor": "apophis doctor --workspace",
"apophis:qualify": "apophis qualify --workspace --profile ci"
}
}
```
### Workspace fan-out
Run verify across all packages:
Use `--workspace` to run verify or doctor across all packages:
```bash
apophis verify --workspace --profile quick --format json
```
Output is package-attributed:
```json
{
"exitCode": 0,
"runs": [
{
"package": "api",
"cwd": "/repo/packages/api",
"artifact": { ... }
},
{
"package": "web",
"cwd": "/repo/packages/web",
"artifact": { ... }
}
]
}
```
### Supported commands
- `apophis verify --workspace`
- `apophis doctor --workspace`
## Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Behavioral / qualification failure |
| 2 | Usage, config, or environment safety violation |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |
See [CLI Reference](cli.md) for workspace output format and exit codes.
+9 -5
View File
@@ -2,6 +2,8 @@
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
## Why APOPHIS Is Good for LLM-Generated Services
Coding agents benefit from:
@@ -18,10 +20,10 @@ Use `apophis init` with a preset:
| Preset | Use Case |
|---|---|
| `safe-ci` | General CI-safe setup |
| `llm-safe` | Ultra-minimal for LLM-generated code |
| `platform-observe` | Observe-mode policy and runtime drift reporting |
| `protocol-lab` | Multi-step flows and stateful testing |
| `safe-ci` | Minimal CI-safe preset (default) |
| `llm-safe` | Minimal preset for LLM-generated codebases |
| `platform-observe` | Production-ready with observe mode |
| `protocol-lab` | Multi-step flow and stateful testing |
```bash
apophis init --preset llm-safe
@@ -108,6 +110,8 @@ export default {
### Route Template with Behavioral Contract
```javascript
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
@@ -134,7 +138,7 @@ app.post('/users', {
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${Date.now()}`;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
+50 -3
View File
@@ -2,6 +2,8 @@
Runtime visibility and drift detection without blocking by default.
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
## What Observe Does
`apophis observe` validates your runtime observe configuration:
@@ -65,14 +67,38 @@ profiles: {
}
```
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema.
The `platform-observe` preset enables sampling. Configure the rate explicitly:
```javascript
profiles: {
'staging-observe': {
mode: 'observe',
preset: 'platform-observe',
routes: [],
sampling: 1.0 // 100% of requests observed
}
}
```
## Staging vs Production
| Environment | Blocking | Sampling | Sink Required |
|---|---|---|---|
| Staging | No (default) | 10% | Yes |
| Production | No (default) | 1% | Yes |
| Staging | No (default) | 100% | Yes |
| Production | No (default) | 100% | Yes |
Default is `1.0` (100%). Configure lower rates for production explicitly:
```javascript
profiles: {
'prod-observe': {
mode: 'observe',
preset: 'platform-observe',
routes: [],
sampling: 0.1 // 10% of requests observed
}
}
```
## `--check-config` Flag
@@ -138,3 +164,24 @@ export default {
}
};
```
## Sink Endpoint Configuration
Configure the reporting sink endpoint in your observe config:
```javascript
observe: {
sink: {
endpoint: 'http://collector.internal:4318'
}
}
```
## Monorepo Validation
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
## Mode Mismatch
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
```
+50 -38
View File
@@ -1,16 +1,18 @@
# APOPHIS Protocol Extensions Specification
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature
## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
## 1. Overview
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
APOPHIS is grounded in [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE) to the core invariant framework.
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
### 1.1 Current Shipped vs Not-Shipped Snapshot
**Shipped in v2.x:**
**Shipped in v2.0.0:**
- `contract({ variants })` for multi-header/media negotiation execution.
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
@@ -166,12 +168,15 @@ jwtExtension({
The JWT extension maintains state across a test run:
```javascript
interface JwtExtensionState {
/** Track seen JTIs for replay detection */
seenJtis: Set<string>
/** Cached decoded JWTs */
decodedCache: Map<string, DecodedJwt>
}
/**
* JWT extension state across a test run.
* @property {Set<string>} seenJtis - Track seen JTIs for replay detection
* @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
*/
const jwtExtensionState = {
seenJtis: new Set(),
decodedCache: new Map()
};
```
### 3.5 Example Contracts
@@ -234,16 +239,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
### 4.4 Implementation
```javascript
interface TimeControl {
/** Advance simulated time by milliseconds */
advance(ms: number): void
/** Set simulated time to specific timestamp */
set(isoString: string): void
/** Get current simulated time */
now(): number
/** Reset to real time */
reset(): void
}
/**
* Time control for deterministic testing.
* @property {function(number): void} advance - Advance simulated time by milliseconds
* @property {function(string): void} set - Set simulated time to specific ISO timestamp
* @property {function(): number} now - Get current simulated time
* @property {function(): void} reset - Reset to real time
*/
const timeControl = {
advance(ms) { /* ... */ },
set(isoString) { /* ... */ },
now() { return Date.now(); },
reset() { /* ... */ }
};
```
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
@@ -288,11 +296,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
Extension state tracks tokens across requests:
```javascript
interface StatefulExtensionState {
seenTokens: Set<string>
consumedTokens: Set<string>
categoryHistory: Map<string, EvalContext> // category -> last context
}
/**
* Stateful extension state tracking tokens across requests.
* @property {Set<string>} seenTokens - Tokens observed in previous requests
* @property {Set<string>} consumedTokens - Tokens that have been consumed
* @property {Map<string, EvalContext>} categoryHistory - category -> last context
*/
const statefulExtensionState = {
seenTokens: new Set(),
consumedTokens: new Set(),
categoryHistory: new Map()
};
```
### 5.4 Example Contracts
@@ -522,14 +536,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
## 14. Implementation Plan
### Phase 1: JWT + Time Control (P0)
**Target**: v1.3.0
### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
**Status**: Complete
**Files**:
- `src/extensions/jwt.ts` — JWT extension implementation
- `src/extensions/time.ts` — Time control extension
- `src/extensions/stateful.ts` — Stateful predicates extension
- `src/test/jwt-extension.test.ts`JWT tests
- `src/test/time-extension.test.ts` — Time control tests
- `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
**Tests**:
- Decode Base64URL claims without verification
@@ -539,27 +553,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
- `now()` predicate with mocked time
- `apophis.time.advance()` in stateful tests
### Phase 2: X.509 + SPIFFE (P1)
**Target**: v1.3.1
### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
**Status**: Complete
**Files**:
- `src/extensions/x509.ts` — X.509 extension
- `src/extensions/spiffe.ts` — SPIFFE extension
- `src/test/x509-extension.test.ts`X.509 tests
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
- `src/test/protocol-extensions.test.ts`Protocol extension tests
### Phase 3: Token Hash + HTTP Signature (P2)
**Target**: v1.3.2
### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
**Status**: Complete
**Files**:
- `src/extensions/token-hash.ts` — Token hash extension
- `src/extensions/http-signature.ts` — HTTP signature extension
- `src/test/token-hash-extension.test.ts`Token hash tests
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
- `src/test/protocol-extensions.test.ts`Protocol extension tests
### Phase 4: Request Context (P2)
**Target**: v1.3.3
### Phase 4: Request Context (P2) — Shipped in v2.0.0
**Status**: Complete
**Files**:
- `src/extensions/request-context.ts` — Request context predicates
- `src/test/request-context-extension.test.ts`Request context tests
- `src/test/protocol-extensions.test.ts`Protocol extension tests
---
+74 -33
View File
@@ -2,6 +2,8 @@
Run scenario, stateful, and chaos checks against non-production Fastify services.
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
## What Qualify Does
`apophis qualify` runs deeper testing than verify:
@@ -49,6 +51,38 @@ profiles: {
}
```
## Scenario Definitions
Scenarios are multi-step flows with capture and rebind:
```javascript
await fastify.apophis.scenario({
name: 'oauth-basic',
steps: [
{
name: 'authorize',
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
expect: ['status:200', 'response_payload(this).code != null'],
capture: { code: 'response_payload(this).code' }
},
{
name: 'token',
request: {
method: 'POST',
url: '/oauth/token',
form: { grant_type: 'authorization_code', code: '$authorize.code' }
},
expect: ['status:200', 'response_payload(this).access_token != null']
}
]
})
```
Scenario behavior:
1. Cookie jar persists `Set-Cookie` values across steps.
2. Step-level `headers.cookie` overrides jar values for that step.
3. `form` sends `application/x-www-form-urlencoded` payloads.
## Stateful Testing
Stateful tests generate sequences of operations and track resources:
@@ -58,7 +92,18 @@ Stateful tests generate sequences of operations and track resources:
3. **Observer**: Read resources (GET)
4. **Destructor**: Remove resources (DELETE)
APOPHIS automatically tracks created resources and cleans them up after testing.
APOPHIS tracks created resources and runs cleanup after test completion.
Run stateful tests via the API:
```javascript
const stateful = await fastify.apophis.stateful({ depth: 'standard', seed: 42 })
console.log('Stateful tests:', stateful.summary)
```
## Route Transparency
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
## Chaos and Adversity
@@ -67,7 +112,9 @@ Chaos testing injects controlled failures:
- **Delay**: Slow responses
- **Error**: Return error status codes
- **Dropout**: Connection failures
- **Corruption**: Malformed response bodies
- **Truncate**: Truncated response bodies
- **Malformed**: Invalid JSON or content-type
- **Field-corrupt**: Random field mutation in response objects
Configure chaos in your preset:
@@ -84,36 +131,6 @@ presets: {
}
```
## Profile Examples
### oauth-nightly
```javascript
profiles: {
'oauth-nightly': {
name: 'oauth-nightly',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
### lifecycle-deep
```javascript
profiles: {
'lifecycle-deep': {
name: 'lifecycle-deep',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
## Non-Prod Boundaries
Qualify mode is gated away from production by default:
@@ -122,7 +139,7 @@ Qualify mode is gated away from production by default:
|---|---|---|---|
| local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | synthetic-only | canary-only |
| staging | enabled with allowlist | enabled | blocked on protected routes |
| production | disabled by default | disabled by default | disabled by default |
## Machine Output for CI
@@ -224,3 +241,27 @@ export default {
}
};
```
## Gate Execution Counts
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
## Zero-Execution Guardrail
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
## `--workspace` Flag
Run qualify across all packages in a monorepo workspace:
```bash
apophis qualify --workspace --profile oauth-nightly
```
## `--generation-profile` Flag
Control test data generation depth independently from the qualification profile:
```bash
apophis qualify --profile oauth-nightly --generation-profile quick
```
+6 -6
View File
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms**
- `Unexpected token` in formula output
- `Unterminated string` in x-ensures clause
- `Unterminated string literal` in x-ensures clause
- `Missing this` in operation call
**Resolution**
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms**
- `Plugin decorator already added`
- `Duplicate route registration`
- `No behavioral contracts found`
- `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
**Resolution**
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
2. Check for multiple imports or plugin registrations in test vs production entry points.
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas.
3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
4. Run `apophis doctor` to verify route discovery matches expectations.
**Prevention**
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
```bash
# Inspect the artifact
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]'
cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
# Replay the exact failure
apophis replay --artifact reports/apophis/verify-<timestamp>.json
apophis replay --artifact reports/apophis/failure-<timestamp>.json
# Filter by error category
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
```
---
+28 -34
View File
@@ -2,15 +2,7 @@
Deterministic contract verification for CI and local development.
## What Verify Does
`apophis verify` runs behavioral contracts against your Fastify routes:
1. Discovers routes from your Fastify app
2. Filters routes by profile config and CLI flags
3. Generates test data from JSON Schema
4. Executes routes and checks `x-ensures` contracts
5. Reports pass/fail with deterministic seed and replay command
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
## When to Use It
@@ -79,6 +71,8 @@ apophis verify --routes "POST /users/*"
apophis verify --profile quick
```
`*` and `?` wildcards are supported in `--routes`.
## `--changed` Flag
Run only routes modified in the current git branch:
@@ -126,6 +120,8 @@ Next
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
```
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
## Machine Output for CI
Use concise formats to reduce log volume in large verify runs:
@@ -137,6 +133,7 @@ Use concise formats to reduce log volume in large verify runs:
```bash
# Extract only failed routes from full ndjson
# Note: route.failed events are only emitted for failures, not passed routes
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
# Write artifact to disk and parse the file instead of stdout
@@ -149,7 +146,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|---|---|
| 0 | All contracts passed |
| 1 | One or more behavioral contracts failed |
| 2 | Config error or no routes matched |
| 2 | Config error, no routes matched, no contracts found, or not a git repo |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |
@@ -158,42 +155,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
```javascript
// apophis.config.js
export default {
mode: 'verify',
profile: 'quick',
profiles: {
quick: {
name: 'quick',
mode: 'verify',
preset: 'safe-ci',
routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
}
},
presets: {
'safe-ci': {
name: 'safe-ci',
depth: 'quick',
timeout: 5000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
timeout: 5000
}
}
};
```
For the full config schema, see [CLI Reference](cli.md).
## Workspace Support
Run verify across all packages in a monorepo workspace:
```bash
apophis verify --workspace --profile quick --format json
```
Output includes per-package pass/fail summaries. Fails if any package fails.
## `--generation-profile` Flag
Control test data generation depth independently from the verification profile:
```bash
apophis verify --profile quick --generation-profile quick
```
+2170
View File
File diff suppressed because it is too large Load Diff
+13 -8
View File
@@ -7,13 +7,13 @@
*
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
*
* Supported inference:
* - required fields response_body(this).field != null
* - minimum (number/integer) response_body(this).field >= N
* - maximum (number/integer) response_body(this).field <= N
* - pattern (string) response_body(this).field matches "..."
* - const response_body(this).field == value
* - enum (small) response_body(this).field == "a" || response_body(this).field == "b"
* Supported inference (all wrapped with status-code guard):
* - required fields response_code(this) == N => response_body(this).field != null
* - minimum (number/integer) response_code(this) == N => response_body(this).field >= N
* - maximum (number/integer) response_code(this) == N => response_body(this).field <= N
* - pattern (string) response_code(this) == N => response_body(this).field matches "..."
* - const response_code(this) == N => response_body(this).field == value
* - enum (small) response_code(this) == N => response_body(this).field == "a" || ...
*
* Not inferred (leave to x-ensures for business logic):
* - minLength/maxLength
@@ -188,7 +188,12 @@ export function inferContractsFromRouteSchema(
const code = parseInt(statusCode, 10)
if (code >= 200 && code < 300) {
const inferred = inferContractsFromResponseSchema(statusSchema)
formulas.push(...inferred)
// Wrap each inferred contract with a status-code guard so it only
// applies when the response actually matches the schema it was
// inferred from. Prevents a 200-schema const from failing on a 404.
for (const formula of inferred) {
formulas.push(`response_code(this) == ${code} => ${formula}`)
}
}
}
+37
View File
@@ -753,3 +753,40 @@ test('integration: route-level x-variants are extracted and executed', async ()
await fastify.close()
}
})
test('integration: inferred contracts are guarded by status code', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {})
fastify.get('/status-guarded', {
schema: {
'x-category': 'observer',
response: {
200: {
type: 'object',
properties: { status: { type: 'string', const: 'success' } },
required: ['status']
},
404: {
type: 'object',
properties: { error: { type: 'string' } },
required: ['error']
}
}
} as Record<string, unknown>
}, async (request, reply) => {
// Return 404 to verify the 200-schema const doesn't fail
reply.status(404)
return { error: 'not found' }
})
await fastify.ready()
const result = await fastify.apophis.contract({ depth: 'quick' })
// Should pass because the inferred const contract is guarded:
// response_code(this) == 200 => response_body(this).status == "success"
// The 404 response doesn't trigger the antecedent, so the implication holds.
assert.strictEqual(result.summary.failed, 0, 'inferred 200-schema const should not fail on 404')
} finally {
await fastify.close()
}
})