docs: final cleanup and accuracy pass before public push

- Fix const inference bug: wrap inferred contracts with status-code guards
- Add integration test for status-guarded contract inference
- Tighten and deduplicate docs across verify, qualify, getting-started, cli
- Fix broken cross-references and TypeScript→JavaScript conversions
- Fix factual errors: license, Date.now(), sampling defaults, cache env
- Add missing features: --workspace, --generation-profile, json-summary formats
- Move stale extension docs (AUTH-RATE-LIMIT-REVISED, HTTP-EXTENSIONS) to attic
- Update PLUGIN_CONTRACTS_SPEC status to Implemented
- Build: clean | Tests: 849 pass, 0 fail
This commit is contained in:
John Dvorak
2026-04-30 11:25:30 -07:00
parent d278c4b105
commit 6c39bd0a6c
19 changed files with 2453 additions and 266 deletions
+13 -3
View File
@@ -12,6 +12,8 @@ apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
## Cross-Route Failure Example ## Cross-Route Failure Example
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another. Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
@@ -19,6 +21,8 @@ Add one behavioral contract next to a route schema. APOPHIS can verify cross-rou
**Route:** **Route:**
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -29,7 +33,7 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
@@ -47,7 +51,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200 response_code(GET /users/{response_body(this).id}) == 200
Observed Observed
GET /users/usr-123 returned 404 GET /users/usr-7d865e returned 404
Why this matters Why this matters
The resource created by POST /users is not retrievable. The resource created by POST /users is not retrievable.
@@ -80,6 +84,9 @@ apophis init --preset safe-ci
# 3. Verify # 3. Verify
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
# 4. Doctor
apophis doctor
``` ```
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough. See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
@@ -87,9 +94,12 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
## Trust and Safety ## Trust and Safety
- **Deterministic replay**: Every failure includes a seed and a one-command replay. - **Deterministic replay**: Every failure includes a seed and a one-command replay.
- **Generation profile aliases**: Control test budget with `--generation-profile quick|standard|deep`.
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines. - **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy. - **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
- **Qualify path gated away from prod**: `qualify` is blocked in production by default. - **Qualify path gated away from prod**: `qualify` is blocked in production by default.
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes. - **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
## LLM-Safe ## LLM-Safe
@@ -116,4 +126,4 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## License ## License
ISC MIT
+1 -1
View File
@@ -1,6 +1,6 @@
# APOPHIS Plugin Contract System Specification # APOPHIS Plugin Contract System Specification
## Status: Active design; target version to be assigned ## Status: Implemented
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins. **Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
+1 -1
View File
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## Operator Resources ## Operator Resources
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps - [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout - [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
## CTAs ## CTAs
+4 -4
View File
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
| Environment | Cache | Reason | | Environment | Cache | Reason |
|-------------|-------|--------| |-------------|-------|--------|
| `production` | Disabled | No file I/O, no cache hits needed | | `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `test` | Disabled | Tests should be deterministic, no cache pollution | | `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
| `development` | Enabled | Speeds up iterative testing | | `development` | Enabled by default | Speeds up iterative testing |
| default | Enabled | Backward compatible | | default | Enabled by default | Backward compatible |
## Cache Invalidation ## Cache Invalidation
+19 -19
View File
@@ -4,7 +4,7 @@ Inject controlled failures into contract tests to validate resilience guarantees
## Usage ## Usage
```typescript ```javascript
const result = await fastify.apophis.contract({ const result = await fastify.apophis.contract({
depth: 'standard', depth: 'standard',
chaos: { chaos: {
@@ -14,7 +14,7 @@ const result = await fastify.apophis.contract({
dropout: { probability: 1 }, dropout: { probability: 1 },
corruption: { probability: 1 }, corruption: { probability: 1 },
}, },
}) });
``` ```
## Event Types ## Event Types
@@ -52,35 +52,35 @@ Mutates response bodies. Tests parsing robustness:
response_body(this).id != null response_body(this).id != null
``` ```
## Content-Type Aware Corruption ## Corruption Strategies
Built-in strategies for common formats: Built-in strategies are content-type agnostic:
| Content-Type | Strategy | Effect | | Strategy | Effect |
|-------------|----------|--------| |----------|--------|
| `application/json` | Truncate or null field | Removes fields or sets random field to null | | `truncate` | Cuts response body short |
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk | | `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
| `text/event-stream` | Event corrupt | Adds malformed SSE line | | `field-corrupt` | Replaces a random field value with corrupted data |
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
| `text/plain` | Truncate | Cuts string in half | Extension strategies can add content-type-specific behavior if needed.
## Custom Corruption via Extensions ## Custom Corruption via Extensions
```typescript ```javascript
const myExtension = { const myExtension = {
name: 'custom-corrupt', name: 'custom-corrupt',
corruptionStrategies: { corruptionStrategies: {
'application/vnd.api+json': (data) => ({ 'application/vnd.api+json': (data) => ({
...data as object, ...data,
corrupted: true, corrupted: true,
}), }),
'text/*': (data) => `CORRUPTED:${String(data)}`, 'text/*': (data) => `CORRUPTED:${String(data)}`,
}, },
} };
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [myExtension], extensions: [myExtension],
}) });
``` ```
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype. Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
@@ -90,7 +90,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run. Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
``` ```
Error: Chaos mode is only available in test environment. Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
``` ```
## Interpreting Results ## Interpreting Results
@@ -123,7 +123,7 @@ Failed tests include chaos events in diagnostics:
## Example: Testing Retry Logic ## Example: Testing Retry Logic
```typescript ```javascript
fastify.get('/data', { fastify.get('/data', {
schema: { schema: {
'x-ensures': [ 'x-ensures': [
@@ -131,7 +131,7 @@ fastify.get('/data', {
'redirect_count(this) <= 3', 'redirect_count(this) <= 3',
], ],
}, },
}, handler) }, handler);
// Test // Test
const result = await fastify.apophis.contract({ const result = await fastify.apophis.contract({
@@ -139,5 +139,5 @@ const result = await fastify.apophis.contract({
probability: 0.2, probability: 0.2,
error: { probability: 1, statusCode: 503 }, error: { probability: 1, statusCode: 503 },
}, },
}) });
``` ```
+33 -4
View File
@@ -10,15 +10,17 @@ Every command accepts these flags:
|---|---|---| |---|---|---|
| `--config <path>` | Config file path | Auto-detect | | `--config <path>` | Config file path | Auto-detect |
| `--profile <name>` | Profile name from config | First profile | | `--profile <name>` | Profile name from config | First profile |
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived | | `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
| `--cwd <path>` | Working directory override | `process.cwd()` | | `--cwd <path>` | Working directory override | `process.cwd()` |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` | | `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` | | `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
| `--quiet` | Suppress non-error output | false | | `--quiet` | Suppress non-error output | false |
| `--verbose` | Enable verbose logging | false | | `--verbose` | Enable verbose logging | false |
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` | | `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
| `--workspace` | Run supported commands across workspace packages | false | | `--workspace` | Run supported commands across workspace packages | false |
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
## Commands ## Commands
### `apophis init` ### `apophis init`
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
| Flag | Description | | Flag | Description |
|---|---| |---|---|
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` | | `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
| `--force` | Overwrite existing files | | `-f, --force` | Overwrite existing files |
| `--noninteractive` | Skip all prompts, require explicit flags | | `--noninteractive` | Skip all prompts, require explicit flags |
**Examples:** **Examples:**
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) | | `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
| `--seed <number>` | Deterministic seed (generated and printed if omitted) | | `--seed <number>` | Deterministic seed (generated and printed if omitted) |
| `--changed` | Filter to git-modified routes only | | `--changed` | Filter to git-modified routes only |
| `--workspace` | Run across all workspace packages |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | | `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
**Examples:** **Examples:**
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|---|---| |---|---|
| `--mode <mode>` | Filter checks to a specific mode | | `--mode <mode>` | Filter checks to a specific mode |
| `--strict` | Treat warnings as failures | | `--strict` | Treat warnings as failures |
| `--workspace` | Run across all workspace packages |
**Checks:** **Checks:**
@@ -210,6 +214,31 @@ apophis migrate --dry-run
apophis migrate --write apophis migrate --write
``` ```
## Common Tasks
### CI workflow with machine output
```bash
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
```
### Monorepo workspace verification
```bash
apophis verify --workspace --profile quick
apophis doctor --workspace
```
### Replay a failure
```bash
apophis replay --artifact reports/apophis/failure-*.json
```
## Gotchas
- `--changed` requires a git repository
- `migrate` defaults to `--dry-run` (safe by default)
- `--workspace` is only supported by `verify` and `doctor` commands
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
## Exit Codes ## Exit Codes
| Code | Meaning | | Code | Meaning |
+2 -1
View File
@@ -1,5 +1,6 @@
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify' import apophisPlugin from 'apophis-fastify'
import crypto from 'crypto'
const fastify = Fastify() const fastify = Fastify()
@@ -40,7 +41,7 @@ fastify.post('/users', {
} }
} }
}, async (req, reply) => { }, async (req, reply) => {
const id = `usr-${Date.now()}` const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
const user = { id, email: req.body.email, name: req.body.name } const user = { id, email: req.body.email, name: req.body.name }
users.set(id, user) users.set(id, user)
reply.status(201) reply.status(201)
+13 -105
View File
@@ -30,6 +30,8 @@ This creates:
Pick one important route. Add an `x-ensures` clause that checks behavior across operations: Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -40,27 +42,20 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
``` ```
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
## Step 4: Run Verify ## Step 4: Run Verify
```bash ```bash
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
APOPHIS will:
1. Discover routes from your Fastify app
2. Filter to `POST /users`
3. Generate test data from the schema
4. Execute the route
5. Check the behavioral contract
6. Print pass/fail, seed, and replay command
## Example Failure ## Example Failure
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it: If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
@@ -100,111 +95,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
## Next Steps ## Next Steps
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"` - Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
- Run all routes: `apophis verify --profile quick` - Run all routes: `apophis verify --profile quick`
- Run only changed routes in CI: `apophis verify --profile ci --changed` - Run only changed routes in CI: `apophis verify --profile ci --changed`
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md) - Requires a git repository.
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md) - Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
## Config Reference ## Config Reference
```javascript For the full configuration reference, see [CLI Reference](cli.md).
// apophis.config.js
export default {
mode: 'verify',
profile: 'quick',
profiles: {
quick: {
name: 'quick',
mode: 'verify',
preset: 'safe-ci',
routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
}
},
presets: {
'safe-ci': {
name: 'safe-ci',
depth: 'quick',
timeout: 5000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
}
}
};
```
## Monorepo Workspaces ## Monorepo Workspaces
APOPHIS supports workspace-wide operations with the `--workspace` flag. Use `--workspace` to run verify or doctor across all packages:
### Root package.json scripts
```json
{
"scripts": {
"apophis:verify": "apophis verify --workspace --profile quick",
"apophis:doctor": "apophis doctor --workspace",
"apophis:qualify": "apophis qualify --workspace --profile ci"
}
}
```
### Workspace fan-out
Run verify across all packages:
```bash ```bash
apophis verify --workspace --profile quick --format json apophis verify --workspace --profile quick --format json
``` ```
Output is package-attributed: See [CLI Reference](cli.md) for workspace output format and exit codes.
```json
{
"exitCode": 0,
"runs": [
{
"package": "api",
"cwd": "/repo/packages/api",
"artifact": { ... }
},
{
"package": "web",
"cwd": "/repo/packages/web",
"artifact": { ... }
}
]
}
```
### Supported commands
- `apophis verify --workspace`
- `apophis doctor --workspace`
## Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Behavioral / qualification failure |
| 2 | Usage, config, or environment safety violation |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |
+7 -5
View File
@@ -18,10 +18,10 @@ Use `apophis init` with a preset:
| Preset | Use Case | | Preset | Use Case |
|---|---| |---|---|
| `safe-ci` | General CI-safe setup | | `safe-ci` | Minimal CI-safe preset (default) |
| `llm-safe` | Ultra-minimal for LLM-generated code | | `llm-safe` | Minimal preset for LLM-generated codebases |
| `platform-observe` | Observe-mode policy and runtime drift reporting | | `platform-observe` | Production-ready with observe mode |
| `protocol-lab` | Multi-step flows and stateful testing | | `protocol-lab` | Multi-step flow and stateful testing |
```bash ```bash
apophis init --preset llm-safe apophis init --preset llm-safe
@@ -108,6 +108,8 @@ export default {
### Route Template with Behavioral Contract ### Route Template with Behavioral Contract
```javascript ```javascript
import crypto from 'crypto';
app.post('/users', { app.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
@@ -134,7 +136,7 @@ app.post('/users', {
} }
}, async (request, reply) => { }, async (request, reply) => {
const { name } = request.body; const { name } = request.body;
const id = `usr-${Date.now()}`; const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201); reply.status(201);
return { id, name }; return { id, name };
}); });
+26 -3
View File
@@ -65,14 +65,16 @@ profiles: {
} }
``` ```
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema. The `platform-observe` preset enables sampling at the preset level.
## Staging vs Production ## Staging vs Production
| Environment | Blocking | Sampling | Sink Required | | Environment | Blocking | Sampling | Sink Required |
|---|---|---|---| |---|---|---|---|
| Staging | No (default) | 10% | Yes | | Staging | No (default) | 100% | Yes |
| Production | No (default) | 1% | Yes | | Production | No (default) | 100% | Yes |
Default is 1.0 (100%). Configure lower rates for production explicitly.
## `--check-config` Flag ## `--check-config` Flag
@@ -138,3 +140,24 @@ export default {
} }
}; };
``` ```
## Sink Endpoint Configuration
Configure the reporting sink endpoint in your observe config:
```javascript
observe: {
sink: {
endpoint: 'http://collector.internal:4318'
}
}
```
## Workspace Support
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages.
## Mode Mismatch
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
```
+48 -38
View File
@@ -1,6 +1,6 @@
# APOPHIS Protocol Extensions Specification # APOPHIS Protocol Extensions Specification
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature ## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
## 1. Overview ## 1. Overview
@@ -10,7 +10,7 @@ Arbiter maintains 58 protocol conformance test files covering 138 behaviors acro
### 1.1 Current Shipped vs Not-Shipped Snapshot ### 1.1 Current Shipped vs Not-Shipped Snapshot
**Shipped in v2.x:** **Shipped in v2.0.0:**
- `contract({ variants })` for multi-header/media negotiation execution. - `contract({ variants })` for multi-header/media negotiation execution.
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows. - `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
@@ -166,12 +166,15 @@ jwtExtension({
The JWT extension maintains state across a test run: The JWT extension maintains state across a test run:
```javascript ```javascript
interface JwtExtensionState { /**
/** Track seen JTIs for replay detection */ * JWT extension state across a test run.
seenJtis: Set<string> * @property {Set<string>} seenJtis - Track seen JTIs for replay detection
/** Cached decoded JWTs */ * @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
decodedCache: Map<string, DecodedJwt> */
} const jwtExtensionState = {
seenJtis: new Set(),
decodedCache: new Map()
};
``` ```
### 3.5 Example Contracts ### 3.5 Example Contracts
@@ -234,16 +237,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
### 4.4 Implementation ### 4.4 Implementation
```javascript ```javascript
interface TimeControl { /**
/** Advance simulated time by milliseconds */ * Time control for deterministic testing.
advance(ms: number): void * @property {function(number): void} advance - Advance simulated time by milliseconds
/** Set simulated time to specific timestamp */ * @property {function(string): void} set - Set simulated time to specific ISO timestamp
set(isoString: string): void * @property {function(): number} now - Get current simulated time
/** Get current simulated time */ * @property {function(): void} reset - Reset to real time
now(): number */
/** Reset to real time */ const timeControl = {
reset(): void advance(ms) { /* ... */ },
} set(isoString) { /* ... */ },
now() { return Date.now(); },
reset() { /* ... */ }
};
``` ```
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time. The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
@@ -288,11 +294,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
Extension state tracks tokens across requests: Extension state tracks tokens across requests:
```javascript ```javascript
interface StatefulExtensionState { /**
seenTokens: Set<string> * Stateful extension state tracking tokens across requests.
consumedTokens: Set<string> * @property {Set<string>} seenTokens - Tokens observed in previous requests
categoryHistory: Map<string, EvalContext> // category -> last context * @property {Set<string>} consumedTokens - Tokens that have been consumed
} * @property {Map<string, EvalContext>} categoryHistory - category -> last context
*/
const statefulExtensionState = {
seenTokens: new Set(),
consumedTokens: new Set(),
categoryHistory: new Map()
};
``` ```
### 5.4 Example Contracts ### 5.4 Example Contracts
@@ -522,14 +534,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
## 14. Implementation Plan ## 14. Implementation Plan
### Phase 1: JWT + Time Control (P0) ### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
**Target**: v1.3.0 **Status**: Complete
**Files**: **Files**:
- `src/extensions/jwt.ts` — JWT extension implementation - `src/extensions/jwt.ts` — JWT extension implementation
- `src/extensions/time.ts` — Time control extension - `src/extensions/time.ts` — Time control extension
- `src/extensions/stateful.ts` — Stateful predicates extension - `src/extensions/stateful.ts` — Stateful predicates extension
- `src/test/jwt-extension.test.ts`JWT tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/time-extension.test.ts` — Time control tests - `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
**Tests**: **Tests**:
- Decode Base64URL claims without verification - Decode Base64URL claims without verification
@@ -539,27 +551,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
- `now()` predicate with mocked time - `now()` predicate with mocked time
- `apophis.time.advance()` in stateful tests - `apophis.time.advance()` in stateful tests
### Phase 2: X.509 + SPIFFE (P1) ### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
**Target**: v1.3.1 **Status**: Complete
**Files**: **Files**:
- `src/extensions/x509.ts` — X.509 extension - `src/extensions/x509.ts` — X.509 extension
- `src/extensions/spiffe.ts` — SPIFFE extension - `src/extensions/spiffe.ts` — SPIFFE extension
- `src/test/x509-extension.test.ts`X.509 tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
### Phase 3: Token Hash + HTTP Signature (P2) ### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
**Target**: v1.3.2 **Status**: Complete
**Files**: **Files**:
- `src/extensions/token-hash.ts` — Token hash extension - `src/extensions/token-hash.ts` — Token hash extension
- `src/extensions/http-signature.ts` — HTTP signature extension - `src/extensions/http-signature.ts` — HTTP signature extension
- `src/test/token-hash-extension.test.ts`Token hash tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
### Phase 4: Request Context (P2) ### Phase 4: Request Context (P2) — Shipped in v2.0.0
**Target**: v1.3.3 **Status**: Complete
**Files**: **Files**:
- `src/extensions/request-context.ts` — Request context predicates - `src/extensions/request-context.ts` — Request context predicates
- `src/test/request-context-extension.test.ts`Request context tests - `src/test/protocol-extensions.test.ts`Protocol extension tests
--- ---
+33 -33
View File
@@ -58,7 +58,11 @@ Stateful tests generate sequences of operations and track resources:
3. **Observer**: Read resources (GET) 3. **Observer**: Read resources (GET)
4. **Destructor**: Remove resources (DELETE) 4. **Destructor**: Remove resources (DELETE)
APOPHIS automatically tracks created resources and cleans them up after testing. APOPHIS tracks created resources and runs cleanup after test completion.
## Route Transparency
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
## Chaos and Adversity ## Chaos and Adversity
@@ -67,7 +71,9 @@ Chaos testing injects controlled failures:
- **Delay**: Slow responses - **Delay**: Slow responses
- **Error**: Return error status codes - **Error**: Return error status codes
- **Dropout**: Connection failures - **Dropout**: Connection failures
- **Corruption**: Malformed response bodies - **Truncate**: Truncated response bodies
- **Malformed**: Invalid JSON or content-type
- **Field-corrupt**: Random field mutation in response objects
Configure chaos in your preset: Configure chaos in your preset:
@@ -84,36 +90,6 @@ presets: {
} }
``` ```
## Profile Examples
### oauth-nightly
```javascript
profiles: {
'oauth-nightly': {
name: 'oauth-nightly',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
### lifecycle-deep
```javascript
profiles: {
'lifecycle-deep': {
name: 'lifecycle-deep',
mode: 'qualify',
preset: 'protocol-lab',
routes: [],
seed: 42
}
}
```
## Non-Prod Boundaries ## Non-Prod Boundaries
Qualify mode is gated away from production by default: Qualify mode is gated away from production by default:
@@ -122,7 +98,7 @@ Qualify mode is gated away from production by default:
|---|---|---|---| |---|---|---|---|
| local | enabled | enabled | enabled | | local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled | | test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | synthetic-only | canary-only | | staging | enabled with allowlist | enabled | blocked on protected routes |
| production | disabled by default | disabled by default | disabled by default | | production | disabled by default | disabled by default | disabled by default |
## Machine Output for CI ## Machine Output for CI
@@ -224,3 +200,27 @@ export default {
} }
}; };
``` ```
## Gate Execution Counts
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
## Zero-Execution Guardrail
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
## `--workspace` Flag
Run qualify across all packages in a monorepo workspace:
```bash
apophis qualify --workspace --profile oauth-nightly
```
## `--generation-profile` Flag
Control test data generation depth independently from the qualification profile:
```bash
apophis qualify --profile oauth-nightly --generation-profile quick
```
+6 -6
View File
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms** **Symptoms**
- `Unexpected token` in formula output - `Unexpected token` in formula output
- `Unterminated string` in x-ensures clause - `Unterminated string literal` in x-ensures clause
- `Missing this` in operation call - `Missing this` in operation call
**Resolution** **Resolution**
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
**Symptoms** **Symptoms**
- `Plugin decorator already added` - `Plugin decorator already added`
- `Duplicate route registration` - `Duplicate route registration`
- `No behavioral contracts found` - `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
**Resolution** **Resolution**
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app. 1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
2. Check for multiple imports or plugin registrations in test vs production entry points. 2. Check for multiple imports or plugin registrations in test vs production entry points.
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas. 3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
4. Run `apophis doctor` to verify route discovery matches expectations. 4. Run `apophis doctor` to verify route discovery matches expectations.
**Prevention** **Prevention**
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
```bash ```bash
# Inspect the artifact # Inspect the artifact
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]' cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
# Replay the exact failure # Replay the exact failure
apophis replay --artifact reports/apophis/verify-<timestamp>.json apophis replay --artifact reports/apophis/failure-<timestamp>.json
# Filter by error category # Filter by error category
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))' cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
``` ```
--- ---
+27 -35
View File
@@ -2,16 +2,6 @@
Deterministic contract verification for CI and local development. Deterministic contract verification for CI and local development.
## What Verify Does
`apophis verify` runs behavioral contracts against your Fastify routes:
1. Discovers routes from your Fastify app
2. Filters routes by profile config and CLI flags
3. Generates test data from JSON Schema
4. Executes routes and checks `x-ensures` contracts
5. Reports pass/fail with deterministic seed and replay command
## When to Use It ## When to Use It
- **Local development**: Quick feedback on behavioral changes - **Local development**: Quick feedback on behavioral changes
@@ -79,6 +69,8 @@ apophis verify --routes "POST /users/*"
apophis verify --profile quick apophis verify --profile quick
``` ```
`*` and `?` wildcards are supported in `--routes`.
## `--changed` Flag ## `--changed` Flag
Run only routes modified in the current git branch: Run only routes modified in the current git branch:
@@ -126,6 +118,8 @@ Next
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
``` ```
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
## Machine Output for CI ## Machine Output for CI
Use concise formats to reduce log volume in large verify runs: Use concise formats to reduce log volume in large verify runs:
@@ -137,6 +131,7 @@ Use concise formats to reduce log volume in large verify runs:
```bash ```bash
# Extract only failed routes from full ndjson # Extract only failed routes from full ndjson
# Note: route.failed events are only emitted for failures, not passed routes
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")' apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
# Write artifact to disk and parse the file instead of stdout # Write artifact to disk and parse the file instead of stdout
@@ -149,7 +144,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|---|---| |---|---|
| 0 | All contracts passed | | 0 | All contracts passed |
| 1 | One or more behavioral contracts failed | | 1 | One or more behavioral contracts failed |
| 2 | Config error or no routes matched | | 2 | Config error, no routes matched, no contracts found, or not a git repo |
| 3 | Internal APOPHIS error | | 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) | | 130 | Interrupted (SIGINT) |
@@ -158,42 +153,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
```javascript ```javascript
// apophis.config.js // apophis.config.js
export default { export default {
mode: 'verify',
profile: 'quick', profile: 'quick',
profiles: { profiles: {
quick: { quick: {
name: 'quick',
mode: 'verify', mode: 'verify',
preset: 'safe-ci', preset: 'safe-ci',
routes: ['POST /users'] routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
} }
}, },
presets: { presets: {
'safe-ci': { 'safe-ci': {
name: 'safe-ci',
depth: 'quick', depth: 'quick',
timeout: 5000, timeout: 5000
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
} }
} }
}; };
``` ```
For the full config schema, see [CLI Reference](cli.md).
## Workspace Support
Run verify across all packages in a monorepo workspace:
```bash
apophis verify --workspace --profile quick --format json
```
Output includes per-package pass/fail summaries. Fails if any package fails.
## `--generation-profile` Flag
Control test data generation depth independently from the verification profile:
```bash
apophis verify --profile quick --generation-profile quick
```
+2170
View File
File diff suppressed because it is too large Load Diff
+13 -8
View File
@@ -7,13 +7,13 @@
* *
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures. * Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
* *
* Supported inference: * Supported inference (all wrapped with status-code guard):
* - required fields response_body(this).field != null * - required fields response_code(this) == N => response_body(this).field != null
* - minimum (number/integer) response_body(this).field >= N * - minimum (number/integer) response_code(this) == N => response_body(this).field >= N
* - maximum (number/integer) response_body(this).field <= N * - maximum (number/integer) response_code(this) == N => response_body(this).field <= N
* - pattern (string) response_body(this).field matches "..." * - pattern (string) response_code(this) == N => response_body(this).field matches "..."
* - const response_body(this).field == value * - const response_code(this) == N => response_body(this).field == value
* - enum (small) response_body(this).field == "a" || response_body(this).field == "b" * - enum (small) response_code(this) == N => response_body(this).field == "a" || ...
* *
* Not inferred (leave to x-ensures for business logic): * Not inferred (leave to x-ensures for business logic):
* - minLength/maxLength * - minLength/maxLength
@@ -188,7 +188,12 @@ export function inferContractsFromRouteSchema(
const code = parseInt(statusCode, 10) const code = parseInt(statusCode, 10)
if (code >= 200 && code < 300) { if (code >= 200 && code < 300) {
const inferred = inferContractsFromResponseSchema(statusSchema) const inferred = inferContractsFromResponseSchema(statusSchema)
formulas.push(...inferred) // Wrap each inferred contract with a status-code guard so it only
// applies when the response actually matches the schema it was
// inferred from. Prevents a 200-schema const from failing on a 404.
for (const formula of inferred) {
formulas.push(`response_code(this) == ${code} => ${formula}`)
}
} }
} }
+37
View File
@@ -753,3 +753,40 @@ test('integration: route-level x-variants are extracted and executed', async ()
await fastify.close() await fastify.close()
} }
}) })
test('integration: inferred contracts are guarded by status code', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {})
fastify.get('/status-guarded', {
schema: {
'x-category': 'observer',
response: {
200: {
type: 'object',
properties: { status: { type: 'string', const: 'success' } },
required: ['status']
},
404: {
type: 'object',
properties: { error: { type: 'string' } },
required: ['error']
}
}
} as Record<string, unknown>
}, async (request, reply) => {
// Return 404 to verify the 200-schema const doesn't fail
reply.status(404)
return { error: 'not found' }
})
await fastify.ready()
const result = await fastify.apophis.contract({ depth: 'quick' })
// Should pass because the inferred const contract is guarded:
// response_code(this) == 200 => response_body(this).status == "success"
// The 404 response doesn't trigger the antecedent, so the implication holds.
assert.strictEqual(result.summary.failed, 0, 'inferred 200-schema const should not fail on 404')
} finally {
await fastify.close()
}
})