Compare commits

...

3 Commits

Author SHA1 Message Date
John Dvorak bf7376b5ad docs: add scenario examples, fix sampling docs, improve pedagogical completeness
- Add actual scenario definition example to qualify.md
- Add stateful test API example to qualify.md
- Fix observe.md sampling section to show explicit config and rates
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:35:38 -07:00
John Dvorak 8d7382417d docs: add paper citations, fix pedagogical issues, improve SKILL.md
- Cite arxiv 2602.23922 (Invariant-Driven Automated Testing) in all major docs
- Add Progressive Complexity section to SKILL.md for LLM guidance
- Fix SKILL.md Fast Start example to use deterministic ID generation
- Fix getting-started.md failure output inconsistency
- Fix auth-patterns.md TypeScript syntax in JS doc
- Fix fastify-structure.md Date.now() in test helper
- Fix observe.md misleading workspace heading
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:34:00 -07:00
John Dvorak 6c39bd0a6c docs: final cleanup and accuracy pass before public push
- Fix const inference bug: wrap inferred contracts with status-code guards
- Add integration test for status-guarded contract inference
- Tighten and deduplicate docs across verify, qualify, getting-started, cli
- Fix broken cross-references and TypeScript→JavaScript conversions
- Fix factual errors: license, Date.now(), sampling defaults, cache env
- Add missing features: --workspace, --generation-profile, json-summary formats
- Move stale extension docs (AUTH-RATE-LIMIT-REVISED, HTTP-EXTENSIONS) to attic
- Update PLUGIN_CONTRACTS_SPEC status to Implemented
- Build: clean | Tests: 849 pass, 0 fail
2026-04-30 11:25:30 -07:00
22 changed files with 2564 additions and 269 deletions
+15 -3
View File
@@ -4,6 +4,8 @@ Behavioral confidence for Fastify services.
APOPHIS checks whether route behavior holds across operations, states, and protocol flows. APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
Supported Node.js versions: 20.x and 22.x. Supported Node.js versions: 20.x and 22.x.
```bash ```bash
@@ -12,6 +14,8 @@ apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
## Cross-Route Failure Example ## Cross-Route Failure Example
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another. Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
@@ -19,6 +23,8 @@ Add one behavioral contract next to a route schema. APOPHIS can verify cross-rou
**Route:** **Route:**
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -29,7 +35,7 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
@@ -47,7 +53,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200 response_code(GET /users/{response_body(this).id}) == 200
Observed Observed
GET /users/usr-123 returned 404 GET /users/usr-7d865e returned 404
Why this matters Why this matters
The resource created by POST /users is not retrievable. The resource created by POST /users is not retrievable.
@@ -80,6 +86,9 @@ apophis init --preset safe-ci
# 3. Verify # 3. Verify
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
# 4. Doctor
apophis doctor
``` ```
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough. See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
@@ -87,9 +96,12 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
## Trust and Safety ## Trust and Safety
- **Deterministic replay**: Every failure includes a seed and a one-command replay. - **Deterministic replay**: Every failure includes a seed and a one-command replay.
- **Generation profile aliases**: Control test budget with `--generation-profile quick|standard|deep`.
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines. - **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy. - **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
- **Qualify path gated away from prod**: `qualify` is blocked in production by default. - **Qualify path gated away from prod**: `qualify` is blocked in production by default.
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes. - **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
## LLM-Safe ## LLM-Safe
@@ -116,4 +128,4 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## License ## License
ISC MIT
+30 -1
View File
@@ -7,6 +7,8 @@ description: Use this skill when adding or improving APOPHIS contract-driven tes
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing. APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
## When To Use ## When To Use
Use this skill when the operator asks to: Use this skill when the operator asks to:
@@ -76,6 +78,7 @@ When entering a Fastify codebase:
import Fastify from 'fastify' import Fastify from 'fastify'
import swagger from '@fastify/swagger' import swagger from '@fastify/swagger'
import apophis from 'apophis-fastify' import apophis from 'apophis-fastify'
import crypto from 'crypto'
const app = Fastify() const app = Fastify()
await app.register(swagger) await app.register(swagger)
@@ -114,8 +117,9 @@ app.post('/users', {
} }
} }
}, async (req, reply) => { }, async (req, reply) => {
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
reply.status(201) reply.status(201)
return { id: 'usr-1', ...req.body } return { id, ...req.body }
}) })
await app.ready() await app.ready()
@@ -350,6 +354,31 @@ Operator framing:
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state. > The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
## Progressive Complexity
Start simple and add depth only where it pays off:
**Level 1 — Status and shape**: Every route gets an expected status code and key field existence.
```apostl
status:201
response_body(this).id != null
```
**Level 2 — Cross-route behavior**: Constructors check retrievability; mutators check persistence.
```apostl
response_code(GET /users/{response_body(this).id}) == 200
response_body(GET /users/{response_body(this).id}).email == request_body(this).email
```
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
```apostl
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
```
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
Add level 2 before level 4. Do not skip level 2 for resource APIs.
## Anti-Patterns ## Anti-Patterns
Do not: Do not:
+1 -1
View File
@@ -1,6 +1,6 @@
# APOPHIS Plugin Contract System Specification # APOPHIS Plugin Contract System Specification
## Status: Active design; target version to be assigned ## Status: Implemented
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins. **Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
+1 -1
View File
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## Operator Resources ## Operator Resources
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps - [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout - [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
## CTAs ## CTAs
+1 -1
View File
@@ -144,7 +144,7 @@ See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
`getToken` runs per request. Handle refresh inline: `getToken` runs per request. Handle refresh inline:
```javascript ```javascript
let cachedToken: string | null = null let cachedToken = null
const auth = createAuthExtension({ const auth = createAuthExtension({
name: 'jwt-with-refresh', name: 'jwt-with-refresh',
+4 -4
View File
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
| Environment | Cache | Reason | | Environment | Cache | Reason |
|-------------|-------|--------| |-------------|-------|--------|
| `production` | Disabled | No file I/O, no cache hits needed | | `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `test` | Disabled | Tests should be deterministic, no cache pollution | | `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `development` | Enabled | Speeds up iterative testing | | `development` | Enabled by default | Speeds up iterative testing |
| default | Enabled | Backward compatible | | default | Enabled by default | Backward compatible |
## Cache Invalidation ## Cache Invalidation
+21 -19
View File
@@ -2,9 +2,11 @@
Inject controlled failures into contract tests to validate resilience guarantees. Inject controlled failures into contract tests to validate resilience guarantees.
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
## Usage ## Usage
```typescript ```javascript
const result = await fastify.apophis.contract({ const result = await fastify.apophis.contract({
depth: 'standard', depth: 'standard',
chaos: { chaos: {
@@ -14,7 +16,7 @@ const result = await fastify.apophis.contract({
dropout: { probability: 1 }, dropout: { probability: 1 },
corruption: { probability: 1 }, corruption: { probability: 1 },
}, },
}) });
``` ```
## Event Types ## Event Types
@@ -52,35 +54,35 @@ Mutates response bodies. Tests parsing robustness:
response_body(this).id != null response_body(this).id != null
``` ```
## Content-Type Aware Corruption ## Corruption Strategies
Built-in strategies for common formats: Built-in strategies are content-type agnostic:
| Content-Type | Strategy | Effect | | Strategy | Effect |
|-------------|----------|--------| |----------|--------|
| `application/json` | Truncate or null field | Removes fields or sets random field to null | | `truncate` | Cuts response body short |
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk | | `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
| `text/event-stream` | Event corrupt | Adds malformed SSE line | | `field-corrupt` | Replaces a random field value with corrupted data |
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
| `text/plain` | Truncate | Cuts string in half | Extension strategies can add content-type-specific behavior if needed.
## Custom Corruption via Extensions ## Custom Corruption via Extensions
```typescript ```javascript
const myExtension = { const myExtension = {
name: 'custom-corrupt', name: 'custom-corrupt',
corruptionStrategies: { corruptionStrategies: {
'application/vnd.api+json': (data) => ({ 'application/vnd.api+json': (data) => ({
...data as object, ...data,
corrupted: true, corrupted: true,
}), }),
'text/*': (data) => `CORRUPTED:${String(data)}`, 'text/*': (data) => `CORRUPTED:${String(data)}`,
}, },
} };
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [myExtension], extensions: [myExtension],
}) });
``` ```
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype. Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
@@ -90,7 +92,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run. Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
``` ```
Error: Chaos mode is only available in test environment. Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
``` ```
## Interpreting Results ## Interpreting Results
@@ -123,7 +125,7 @@ Failed tests include chaos events in diagnostics:
## Example: Testing Retry Logic ## Example: Testing Retry Logic
```typescript ```javascript
fastify.get('/data', { fastify.get('/data', {
schema: { schema: {
'x-ensures': [ 'x-ensures': [
@@ -131,7 +133,7 @@ fastify.get('/data', {
'redirect_count(this) <= 3', 'redirect_count(this) <= 3',
], ],
}, },
}, handler) }, handler);
// Test // Test
const result = await fastify.apophis.contract({ const result = await fastify.apophis.contract({
@@ -139,5 +141,5 @@ const result = await fastify.apophis.contract({
probability: 0.2, probability: 0.2,
error: { probability: 1, statusCode: 503 }, error: { probability: 1, statusCode: 503 },
}, },
}) });
``` ```
+33 -4
View File
@@ -10,15 +10,17 @@ Every command accepts these flags:
|---|---|---| |---|---|---|
| `--config <path>` | Config file path | Auto-detect | | `--config <path>` | Config file path | Auto-detect |
| `--profile <name>` | Profile name from config | First profile | | `--profile <name>` | Profile name from config | First profile |
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived | | `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
| `--cwd <path>` | Working directory override | `process.cwd()` | | `--cwd <path>` | Working directory override | `process.cwd()` |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` | | `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` | | `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
| `--quiet` | Suppress non-error output | false | | `--quiet` | Suppress non-error output | false |
| `--verbose` | Enable verbose logging | false | | `--verbose` | Enable verbose logging | false |
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` | | `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
| `--workspace` | Run supported commands across workspace packages | false | | `--workspace` | Run supported commands across workspace packages | false |
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
## Commands ## Commands
### `apophis init` ### `apophis init`
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
| Flag | Description | | Flag | Description |
|---|---| |---|---|
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` | | `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
| `--force` | Overwrite existing files | | `-f, --force` | Overwrite existing files |
| `--noninteractive` | Skip all prompts, require explicit flags | | `--noninteractive` | Skip all prompts, require explicit flags |
**Examples:** **Examples:**
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) | | `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
| `--seed <number>` | Deterministic seed (generated and printed if omitted) | | `--seed <number>` | Deterministic seed (generated and printed if omitted) |
| `--changed` | Filter to git-modified routes only | | `--changed` | Filter to git-modified routes only |
| `--workspace` | Run across all workspace packages |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | | `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
**Examples:** **Examples:**
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|---|---| |---|---|
| `--mode <mode>` | Filter checks to a specific mode | | `--mode <mode>` | Filter checks to a specific mode |
| `--strict` | Treat warnings as failures | | `--strict` | Treat warnings as failures |
| `--workspace` | Run across all workspace packages |
**Checks:** **Checks:**
@@ -210,6 +214,31 @@ apophis migrate --dry-run
apophis migrate --write apophis migrate --write
``` ```
## Common Tasks
### CI workflow with machine output
```bash
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
```
### Monorepo workspace verification
```bash
apophis verify --workspace --profile quick
apophis doctor --workspace
```
### Replay a failure
```bash
apophis replay --artifact reports/apophis/failure-*.json
```
## Gotchas
- `--changed` requires a git repository
- `migrate` defaults to `--dry-run` (safe by default)
- `--workspace` is only supported by `verify` and `doctor` commands
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
## Exit Codes ## Exit Codes
| Code | Meaning | | Code | Meaning |
+2 -1
View File
@@ -1,5 +1,6 @@
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify' import apophisPlugin from 'apophis-fastify'
import crypto from 'crypto'
const fastify = Fastify() const fastify = Fastify()
@@ -40,7 +41,7 @@ fastify.post('/users', {
} }
} }
}, async (req, reply) => { }, async (req, reply) => {
const id = `usr-${Date.now()}` const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
const user = { id, email: req.body.email, name: req.body.name } const user = { id, email: req.body.email, name: req.body.name }
users.set(id, user) users.set(id, user)
reply.status(201) reply.status(201)
+3 -1
View File
@@ -374,8 +374,10 @@ import { mkdirSync, rmSync } from 'fs'
import { tmpdir } from 'os' import { tmpdir } from 'os'
import { join } from 'path' import { join } from 'path'
let testCounter = 0
export function createTestWorkspace() { export function createTestWorkspace() {
const dir = join(tmpdir(), `apophis-test-${Date.now()}`) const dir = join(tmpdir(), `apophis-test-${++testCounter}`)
mkdirSync(dir, { recursive: true }) mkdirSync(dir, { recursive: true })
return { return {
+16 -106
View File
@@ -2,6 +2,8 @@
Get from install to your first behavioral bug in 10 minutes. Get from install to your first behavioral bug in 10 minutes.
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
## Prerequisites ## Prerequisites
- Node.js 20.x or 22.x - Node.js 20.x or 22.x
@@ -30,6 +32,8 @@ This creates:
Pick one important route. Add an `x-ensures` clause that checks behavior across operations: Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -40,27 +44,20 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
``` ```
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
## Step 4: Run Verify ## Step 4: Run Verify
```bash ```bash
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
APOPHIS will:
1. Discover routes from your Fastify app
2. Filter to `POST /users`
3. Generate test data from the schema
4. Execute the route
5. Check the behavioral contract
6. Print pass/fail, seed, and replay command
## Example Failure ## Example Failure
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it: If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
@@ -75,7 +72,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200 response_code(GET /users/{response_body(this).id}) == 200
Observed Observed
GET /users/usr-123 returned 404 GET /users/usr-7d865e returned 404
Why this matters Why this matters
The resource created by POST /users is not retrievable. The resource created by POST /users is not retrievable.
@@ -100,111 +97,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
## Next Steps ## Next Steps
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"` - Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
- Run all routes: `apophis verify --profile quick` - Run all routes: `apophis verify --profile quick`
- Run only changed routes in CI: `apophis verify --profile ci --changed` - Run only changed routes in CI: `apophis verify --profile ci --changed`
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md) - Requires a git repository.
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md) - Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
## Config Reference ## Config Reference
```javascript For the full configuration reference, see [CLI Reference](cli.md).
// apophis.config.js
export default {
mode: 'verify',
profile: 'quick',
profiles: {
quick: {
name: 'quick',
mode: 'verify',
preset: 'safe-ci',
routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
}
},
presets: {
'safe-ci': {
name: 'safe-ci',
depth: 'quick',
timeout: 5000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
}
}
};
```
## Monorepo Workspaces ## Monorepo Workspaces
APOPHIS supports workspace-wide operations with the `--workspace` flag. Use `--workspace` to run verify or doctor across all packages:
### Root package.json scripts
```json
{
"scripts": {
"apophis:verify": "apophis verify --workspace --profile quick",
"apophis:doctor": "apophis doctor --workspace",
"apophis:qualify": "apophis qualify --workspace --profile ci"
}
}
```
### Workspace fan-out
Run verify across all packages:
```bash ```bash
apophis verify --workspace --profile quick --format json apophis verify --workspace --profile quick --format json
``` ```
Output is package-attributed: See [CLI Reference](cli.md) for workspace output format and exit codes.
```json
{
"exitCode": 0,
"runs": [
{
"package": "api",
"cwd": "/repo/packages/api",
"artifact": { ... }
},
{
"package": "web",
"cwd": "/repo/packages/web",
"artifact": { ... }
}
]
}
```
### Supported commands
- `apophis verify --workspace`
- `apophis doctor --workspace`
## Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Behavioral / qualification failure |
| 2 | Usage, config, or environment safety violation |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |
+9 -5
View File
@@ -2,6 +2,8 @@
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services. APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
## Why APOPHIS Is Good for LLM-Generated Services ## Why APOPHIS Is Good for LLM-Generated Services
Coding agents benefit from: Coding agents benefit from:
@@ -18,10 +20,10 @@ Use `apophis init` with a preset:
| Preset | Use Case | | Preset | Use Case |
|---|---| |---|---|
| `safe-ci` | General CI-safe setup | | `safe-ci` | Minimal CI-safe preset (default) |
| `llm-safe` | Ultra-minimal for LLM-generated code | | `llm-safe` | Minimal preset for LLM-generated codebases |
| `platform-observe` | Observe-mode policy and runtime drift reporting | | `platform-observe` | Production-ready with observe mode |
| `protocol-lab` | Multi-step flows and stateful testing | | `protocol-lab` | Multi-step flow and stateful testing |
```bash ```bash
apophis init --preset llm-safe apophis init --preset llm-safe
@@ -108,6 +110,8 @@ export default {
### Route Template with Behavioral Contract ### Route Template with Behavioral Contract
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -134,7 +138,7 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
+50 -3
View File
@@ -2,6 +2,8 @@
Runtime visibility and drift detection without blocking by default. Runtime visibility and drift detection without blocking by default.
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
## What Observe Does ## What Observe Does
`apophis observe` validates your runtime observe configuration: `apophis observe` validates your runtime observe configuration:
@@ -65,14 +67,38 @@ profiles: {
} }
``` ```
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema. The `platform-observe` preset enables sampling. Configure the rate explicitly:
```javascript
profiles: {
'staging-observe': {
mode: 'observe',
preset: 'platform-observe',
routes: [],
sampling: 1.0 // 100% of requests observed
}
}
```
## Staging vs Production ## Staging vs Production
| Environment | Blocking | Sampling | Sink Required | | Environment | Blocking | Sampling | Sink Required |
|---|---|---|---| |---|---|---|---|
| Staging | No (default) | 10% | Yes | | Staging | No (default) | 100% | Yes |
| Production | No (default) | 1% | Yes | | Production | No (default) | 100% | Yes |
Default is `1.0` (100%). Configure lower rates for production explicitly:
```javascript
profiles: {
'prod-observe': {
mode: 'observe',
preset: 'platform-observe',
routes: [],
sampling: 0.1 // 10% of requests observed
}
}
```
## `--check-config` Flag ## `--check-config` Flag
@@ -138,3 +164,24 @@ export default {
} }
}; };
``` ```
## Sink Endpoint Configuration
Configure the reporting sink endpoint in your observe config:
```javascript
observe: {
sink: {
endpoint: 'http://collector.internal:4318'
}
}
```
## Monorepo Validation
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
## Mode Mismatch
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
```
+50 -38
View File
@@ -1,16 +1,18 @@
# APOPHIS Protocol Extensions Specification # APOPHIS Protocol Extensions Specification
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature ## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
## 1. Overview ## 1. Overview
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols. This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
APOPHIS is grounded in [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE) to the core invariant framework.
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation. Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
### 1.1 Current Shipped vs Not-Shipped Snapshot ### 1.1 Current Shipped vs Not-Shipped Snapshot
**Shipped in v2.x:** **Shipped in v2.0.0:**
- `contract({ variants })` for multi-header/media negotiation execution. - `contract({ variants })` for multi-header/media negotiation execution.
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows. - `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
@@ -166,12 +168,15 @@ jwtExtension({
The JWT extension maintains state across a test run: The JWT extension maintains state across a test run:
```javascript ```javascript
interface JwtExtensionState { /**
/** Track seen JTIs for replay detection */ * JWT extension state across a test run.
seenJtis: Set<string> * @property {Set<string>} seenJtis - Track seen JTIs for replay detection
/** Cached decoded JWTs */ * @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
decodedCache: Map<string, DecodedJwt> */
} const jwtExtensionState = {
seenJtis: new Set(),
decodedCache: new Map()
};
``` ```
### 3.5 Example Contracts ### 3.5 Example Contracts
@@ -234,16 +239,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
### 4.4 Implementation ### 4.4 Implementation
```javascript ```javascript
interface TimeControl { /**
/** Advance simulated time by milliseconds */ * Time control for deterministic testing.
advance(ms: number): void * @property {function(number): void} advance - Advance simulated time by milliseconds
/** Set simulated time to specific timestamp */ * @property {function(string): void} set - Set simulated time to specific ISO timestamp
set(isoString: string): void * @property {function(): number} now - Get current simulated time
/** Get current simulated time */ * @property {function(): void} reset - Reset to real time
now(): number */
/** Reset to real time */ const timeControl = {
reset(): void advance(ms) { /* ... */ },
} set(isoString) { /* ... */ },
now() { return Date.now(); },
reset() { /* ... */ }
};
``` ```
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time. The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
@@ -288,11 +296,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
Extension state tracks tokens across requests: Extension state tracks tokens across requests:
```javascript ```javascript
interface StatefulExtensionState { /**
seenTokens: Set<string> * Stateful extension state tracking tokens across requests.
consumedTokens: Set<string> * @property {Set<string>} seenTokens - Tokens observed in previous requests
categoryHistory: Map<string, EvalContext> // category -> last context * @property {Set<string>} consumedTokens - Tokens that have been consumed
} * @property {Map<string, EvalContext>} categoryHistory - category -> last context
*/
const statefulExtensionState = {
seenTokens: new Set(),
consumedTokens: new Set(),
categoryHistory: new Map()
};
``` ```
### 5.4 Example Contracts ### 5.4 Example Contracts
@@ -522,14 +536,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
## 14. Implementation Plan ## 14. Implementation Plan
### Phase 1: JWT + Time Control (P0) ### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
**Target**: v1.3.0 **Status**: Complete
**Files**: **Files**:
- `src/extensions/jwt.ts` — JWT extension implementation - `src/extensions/jwt.ts` — JWT extension implementation
- `src/extensions/time.ts` — Time control extension - `src/extensions/time.ts` — Time control extension
- `src/extensions/stateful.ts` — Stateful predicates extension - `src/extensions/stateful.ts` — Stateful predicates extension
- `src/test/jwt-extension.test.ts`JWT tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/time-extension.test.ts` — Time control tests - `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
**Tests**: **Tests**:
- Decode Base64URL claims without verification - Decode Base64URL claims without verification
@@ -539,27 +553,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
- `now()` predicate with mocked time - `now()` predicate with mocked time
- `apophis.time.advance()` in stateful tests - `apophis.time.advance()` in stateful tests
### Phase 2: X.509 + SPIFFE (P1) ### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
**Target**: v1.3.1 **Status**: Complete
**Files**: **Files**:
- `src/extensions/x509.ts` — X.509 extension - `src/extensions/x509.ts` — X.509 extension
- `src/extensions/spiffe.ts` — SPIFFE extension - `src/extensions/spiffe.ts` — SPIFFE extension
- `src/test/x509-extension.test.ts`X.509 tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
### Phase 3: Token Hash + HTTP Signature (P2) ### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
**Target**: v1.3.2 **Status**: Complete
**Files**: **Files**:
- `src/extensions/token-hash.ts` — Token hash extension - `src/extensions/token-hash.ts` — Token hash extension
- `src/extensions/http-signature.ts` — HTTP signature extension - `src/extensions/http-signature.ts` — HTTP signature extension
- `src/test/token-hash-extension.test.ts`Token hash tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
### Phase 4: Request Context (P2) ### Phase 4: Request Context (P2) — Shipped in v2.0.0
**Target**: v1.3.3 **Status**: Complete
**Files**: **Files**:
- `src/extensions/request-context.ts` — Request context predicates - `src/extensions/request-context.ts` — Request context predicates
- `src/test/request-context-extension.test.ts`Request context tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
--- ---
+74 -33
View File
@@ -2,6 +2,8 @@
Run scenario, stateful, and chaos checks against non-production Fastify services. Run scenario, stateful, and chaos checks against non-production Fastify services.
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
## What Qualify Does ## What Qualify Does
`apophis qualify` runs deeper testing than verify: `apophis qualify` runs deeper testing than verify:
@@ -49,6 +51,38 @@ profiles: {
} }
``` ```
## Scenario Definitions
Scenarios are multi-step flows with capture and rebind:
```javascript
await fastify.apophis.scenario({
name: 'oauth-basic',
steps: [
{
name: 'authorize',
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
expect: ['status:200', 'response_payload(this).code != null'],
capture: { code: 'response_payload(this).code' }
},
{
name: 'token',
request: {
method: 'POST',
url: '/oauth/token',
form: { grant_type: 'authorization_code', code: '$authorize.code' }
},
expect: ['status:200', 'response_payload(this).access_token != null']
}
]
})
```
Scenario behavior:
1. Cookie jar persists `Set-Cookie` values across steps.
2. Step-level `headers.cookie` overrides jar values for that step.
3. `form` sends `application/x-www-form-urlencoded` payloads.
## Stateful Testing ## Stateful Testing
Stateful tests generate sequences of operations and track resources: Stateful tests generate sequences of operations and track resources:
@@ -58,7 +92,18 @@ Stateful tests generate sequences of operations and track resources:
3. **Observer**: Read resources (GET) 3. **Observer**: Read resources (GET)
4. **Destructor**: Remove resources (DELETE) 4. **Destructor**: Remove resources (DELETE)
APOPHIS automatically tracks created resources and cleans them up after testing. APOPHIS tracks created resources and runs cleanup after test completion.
Run stateful tests via the API:
```javascript
const stateful = await fastify.apophis.stateful({ depth: 'standard', seed: 42 })
console.log('Stateful tests:', stateful.summary)
```
## Route Transparency
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
## Chaos and Adversity ## Chaos and Adversity
@@ -67,7 +112,9 @@ Chaos testing injects controlled failures:
- **Delay**: Slow responses - **Delay**: Slow responses
- **Error**: Return error status codes - **Error**: Return error status codes
- **Dropout**: Connection failures - **Dropout**: Connection failures
- **Corruption**: Malformed response bodies - **Truncate**: Truncated response bodies
- **Malformed**: Invalid JSON or content-type
- **Field-corrupt**: Random field mutation in response objects
Configure chaos in your preset: Configure chaos in your preset:
@@ -84,36 +131,6 @@ presets: {
} }
``` ```
## Profile Examples
### oauth-nightly
```javascript
profiles: {
'oauth-nightly': {
name: 'oauth-nightly',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
### lifecycle-deep
```javascript
profiles: {
'lifecycle-deep': {
name: 'lifecycle-deep',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
## Non-Prod Boundaries ## Non-Prod Boundaries
Qualify mode is gated away from production by default: Qualify mode is gated away from production by default:
@@ -122,7 +139,7 @@ Qualify mode is gated away from production by default:
|---|---|---|---| |---|---|---|---|
| local | enabled | enabled | enabled | | local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled | | test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | synthetic-only | canary-only | | staging | enabled with allowlist | enabled | blocked on protected routes |
| production | disabled by default | disabled by default | disabled by default | | production | disabled by default | disabled by default | disabled by default |
## Machine Output for CI ## Machine Output for CI
@@ -224,3 +241,27 @@ export default {
} }
}; };
``` ```
## Gate Execution Counts
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
## Zero-Execution Guardrail
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
## `--workspace` Flag
Run qualify across all packages in a monorepo workspace:
```bash
apophis qualify --workspace --profile oauth-nightly
```
## `--generation-profile` Flag
Control test data generation depth independently from the qualification profile:
```bash
apophis qualify --profile oauth-nightly --generation-profile quick
```
+6 -6
View File
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms** **Symptoms**
- `Unexpected token` in formula output - `Unexpected token` in formula output
- `Unterminated string` in x-ensures clause - `Unterminated string literal` in x-ensures clause
- `Missing this` in operation call - `Missing this` in operation call
**Resolution** **Resolution**
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms** **Symptoms**
- `Plugin decorator already added` - `Plugin decorator already added`
- `Duplicate route registration` - `Duplicate route registration`
- `No behavioral contracts found` - `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
**Resolution** **Resolution**
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app. 1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
2. Check for multiple imports or plugin registrations in test vs production entry points. 2. Check for multiple imports or plugin registrations in test vs production entry points.
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas. 3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
4. Run `apophis doctor` to verify route discovery matches expectations. 4. Run `apophis doctor` to verify route discovery matches expectations.
**Prevention** **Prevention**
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
```bash ```bash
# Inspect the artifact # Inspect the artifact
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]' cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
# Replay the exact failure # Replay the exact failure
apophis replay --artifact reports/apophis/verify-<timestamp>.json apophis replay --artifact reports/apophis/failure-<timestamp>.json
# Filter by error category # Filter by error category
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))' cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
``` ```
--- ---
+28 -34
View File
@@ -2,15 +2,7 @@
Deterministic contract verification for CI and local development. Deterministic contract verification for CI and local development.
## What Verify Does APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
`apophis verify` runs behavioral contracts against your Fastify routes:
1. Discovers routes from your Fastify app
2. Filters routes by profile config and CLI flags
3. Generates test data from JSON Schema
4. Executes routes and checks `x-ensures` contracts
5. Reports pass/fail with deterministic seed and replay command
## When to Use It ## When to Use It
@@ -79,6 +71,8 @@ apophis verify --routes "POST /users/*"
apophis verify --profile quick apophis verify --profile quick
``` ```
`*` and `?` wildcards are supported in `--routes`.
## `--changed` Flag ## `--changed` Flag
Run only routes modified in the current git branch: Run only routes modified in the current git branch:
@@ -126,6 +120,8 @@ Next
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
``` ```
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
## Machine Output for CI ## Machine Output for CI
Use concise formats to reduce log volume in large verify runs: Use concise formats to reduce log volume in large verify runs:
@@ -137,6 +133,7 @@ Use concise formats to reduce log volume in large verify runs:
```bash ```bash
# Extract only failed routes from full ndjson # Extract only failed routes from full ndjson
# Note: route.failed events are only emitted for failures, not passed routes
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")' apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
# Write artifact to disk and parse the file instead of stdout # Write artifact to disk and parse the file instead of stdout
@@ -149,7 +146,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|---|---| |---|---|
| 0 | All contracts passed | | 0 | All contracts passed |
| 1 | One or more behavioral contracts failed | | 1 | One or more behavioral contracts failed |
| 2 | Config error or no routes matched | | 2 | Config error, no routes matched, no contracts found, or not a git repo |
| 3 | Internal APOPHIS error | | 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) | | 130 | Interrupted (SIGINT) |
@@ -158,42 +155,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
```javascript ```javascript
// apophis.config.js // apophis.config.js
export default { export default {
mode: 'verify',
profile: 'quick', profile: 'quick',
profiles: { profiles: {
quick: { quick: {
name: 'quick',
mode: 'verify', mode: 'verify',
preset: 'safe-ci', preset: 'safe-ci',
routes: ['POST /users'] routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
} }
}, },
presets: { presets: {
'safe-ci': { 'safe-ci': {
name: 'safe-ci',
depth: 'quick', depth: 'quick',
timeout: 5000, timeout: 5000
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
} }
} }
}; };
``` ```
For the full config schema, see [CLI Reference](cli.md).
## Workspace Support
Run verify across all packages in a monorepo workspace:
```bash
apophis verify --workspace --profile quick --format json
```
Output includes per-package pass/fail summaries. Fails if any package fails.
## `--generation-profile` Flag
Control test data generation depth independently from the verification profile:
```bash
apophis verify --profile quick --generation-profile quick
```
+2170
View File
File diff suppressed because it is too large Load Diff
+13 -8
View File
@@ -7,13 +7,13 @@
* *
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures. * Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
* *
* Supported inference: * Supported inference (all wrapped with status-code guard):
* - required fields response_body(this).field != null * - required fields response_code(this) == N => response_body(this).field != null
* - minimum (number/integer) response_body(this).field >= N * - minimum (number/integer) response_code(this) == N => response_body(this).field >= N
* - maximum (number/integer) response_body(this).field <= N * - maximum (number/integer) response_code(this) == N => response_body(this).field <= N
* - pattern (string) response_body(this).field matches "..." * - pattern (string) response_code(this) == N => response_body(this).field matches "..."
* - const response_body(this).field == value * - const response_code(this) == N => response_body(this).field == value
* - enum (small) response_body(this).field == "a" || response_body(this).field == "b" * - enum (small) response_code(this) == N => response_body(this).field == "a" || ...
* *
* Not inferred (leave to x-ensures for business logic): * Not inferred (leave to x-ensures for business logic):
* - minLength/maxLength * - minLength/maxLength
@@ -188,7 +188,12 @@ export function inferContractsFromRouteSchema(
const code = parseInt(statusCode, 10) const code = parseInt(statusCode, 10)
if (code >= 200 && code < 300) { if (code >= 200 && code < 300) {
const inferred = inferContractsFromResponseSchema(statusSchema) const inferred = inferContractsFromResponseSchema(statusSchema)
formulas.push(...inferred) // Wrap each inferred contract with a status-code guard so it only
// applies when the response actually matches the schema it was
// inferred from. Prevents a 200-schema const from failing on a 404.
for (const formula of inferred) {
formulas.push(`response_code(this) == ${code} => ${formula}`)
}
} }
} }
+37
View File
@@ -753,3 +753,40 @@ test('integration: route-level x-variants are extracted and executed', async ()
await fastify.close() await fastify.close()
} }
}) })
test('integration: inferred contracts are guarded by status code', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {})
fastify.get('/status-guarded', {
schema: {
'x-category': 'observer',
response: {
200: {
type: 'object',
properties: { status: { type: 'string', const: 'success' } },
required: ['status']
},
404: {
type: 'object',
properties: { error: { type: 'string' } },
required: ['error']
}
}
} as Record<string, unknown>
}, async (request, reply) => {
// Return 404 to verify the 200-schema const doesn't fail
reply.status(404)
return { error: 'not found' }
})
await fastify.ready()
const result = await fastify.apophis.contract({ depth: 'quick' })
// Should pass because the inferred const contract is guarded:
// response_code(this) == 200 => response_body(this).status == "success"
// The 404 response doesn't trigger the antecedent, so the implication holds.
assert.strictEqual(result.summary.failed, 0, 'inferred 200-schema const should not fail on 404')
} finally {
await fastify.close()
}
})