docs: final cleanup and accuracy pass before public push
- Fix const inference bug: wrap inferred contracts with status-code guards - Add integration test for status-guarded contract inference - Tighten and deduplicate docs across verify, qualify, getting-started, cli - Fix broken cross-references and TypeScript→JavaScript conversions - Fix factual errors: license, Date.now(), sampling defaults, cache env - Add missing features: --workspace, --generation-profile, json-summary formats - Move stale extension docs (AUTH-RATE-LIMIT-REVISED, HTTP-EXTENSIONS) to attic - Update PLUGIN_CONTRACTS_SPEC status to Implemented - Build: clean | Tests: 849 pass, 0 fail
This commit is contained in:
John Dvorak
@@ -1,6 +1,6 @@
|
||||
# APOPHIS Plugin Contract System Specification
|
||||
|
||||
## Status: Active design; target version to be assigned
|
||||
## Status: Implemented
|
||||
|
||||
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
||||
## Operator Resources
|
||||
|
||||
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
|
||||
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout
|
||||
- [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
|
||||
|
||||
## CTAs
|
||||
|
||||
|
||||
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
|
||||
|
||||
| Environment | Cache | Reason |
|
||||
|-------------|-------|--------|
|
||||
| `production` | Disabled | No file I/O, no cache hits needed |
|
||||
| `test` | Disabled | Tests should be deterministic, no cache pollution |
|
||||
| `development` | Enabled | Speeds up iterative testing |
|
||||
| default | Enabled | Backward compatible |
|
||||
| `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `development` | Enabled by default | Speeds up iterative testing |
|
||||
| default | Enabled by default | Backward compatible |
|
||||
|
||||
## Cache Invalidation
|
||||
|
||||
|
||||
+19
-19
@@ -4,7 +4,7 @@ Inject controlled failures into contract tests to validate resilience guarantees
|
||||
|
||||
## Usage
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const result = await fastify.apophis.contract({
|
||||
depth: 'standard',
|
||||
chaos: {
|
||||
@@ -14,7 +14,7 @@ const result = await fastify.apophis.contract({
|
||||
dropout: { probability: 1 },
|
||||
corruption: { probability: 1 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
## Event Types
|
||||
@@ -52,35 +52,35 @@ Mutates response bodies. Tests parsing robustness:
|
||||
response_body(this).id != null
|
||||
```
|
||||
|
||||
## Content-Type Aware Corruption
|
||||
## Corruption Strategies
|
||||
|
||||
Built-in strategies for common formats:
|
||||
Built-in strategies are content-type agnostic:
|
||||
|
||||
| Content-Type | Strategy | Effect |
|
||||
|-------------|----------|--------|
|
||||
| `application/json` | Truncate or null field | Removes fields or sets random field to null |
|
||||
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk |
|
||||
| `text/event-stream` | Event corrupt | Adds malformed SSE line |
|
||||
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
|
||||
| `text/plain` | Truncate | Cuts string in half |
|
||||
| Strategy | Effect |
|
||||
|----------|--------|
|
||||
| `truncate` | Cuts response body short |
|
||||
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
|
||||
| `field-corrupt` | Replaces a random field value with corrupted data |
|
||||
|
||||
Extension strategies can add content-type-specific behavior if needed.
|
||||
|
||||
## Custom Corruption via Extensions
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const myExtension = {
|
||||
name: 'custom-corrupt',
|
||||
corruptionStrategies: {
|
||||
'application/vnd.api+json': (data) => ({
|
||||
...data as object,
|
||||
...data,
|
||||
corrupted: true,
|
||||
}),
|
||||
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
||||
},
|
||||
}
|
||||
};
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [myExtension],
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
||||
@@ -90,7 +90,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
|
||||
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
||||
|
||||
```
|
||||
Error: Chaos mode is only available in test environment.
|
||||
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
|
||||
```
|
||||
|
||||
## Interpreting Results
|
||||
@@ -123,7 +123,7 @@ Failed tests include chaos events in diagnostics:
|
||||
|
||||
## Example: Testing Retry Logic
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
fastify.get('/data', {
|
||||
schema: {
|
||||
'x-ensures': [
|
||||
@@ -131,7 +131,7 @@ fastify.get('/data', {
|
||||
'redirect_count(this) <= 3',
|
||||
],
|
||||
},
|
||||
}, handler)
|
||||
}, handler);
|
||||
|
||||
// Test
|
||||
const result = await fastify.apophis.contract({
|
||||
@@ -139,5 +139,5 @@ const result = await fastify.apophis.contract({
|
||||
probability: 0.2,
|
||||
error: { probability: 1, statusCode: 503 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
+33
-4
@@ -10,15 +10,17 @@ Every command accepts these flags:
|
||||
|---|---|---|
|
||||
| `--config <path>` | Config file path | Auto-detect |
|
||||
| `--profile <name>` | Profile name from config | First profile |
|
||||
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived |
|
||||
| `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
|
||||
| `--cwd <path>` | Working directory override | `process.cwd()` |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
|
||||
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
|
||||
| `--quiet` | Suppress non-error output | false |
|
||||
| `--verbose` | Enable verbose logging | false |
|
||||
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` |
|
||||
| `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
|
||||
| `--workspace` | Run supported commands across workspace packages | false |
|
||||
|
||||
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
|
||||
|
||||
## Commands
|
||||
|
||||
### `apophis init`
|
||||
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `--force` | Overwrite existing files |
|
||||
| `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `-f, --force` | Overwrite existing files |
|
||||
| `--noninteractive` | Skip all prompts, require explicit flags |
|
||||
|
||||
**Examples:**
|
||||
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
|
||||
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||
| `--changed` | Filter to git-modified routes only |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
||||
|
||||
**Examples:**
|
||||
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|
||||
|---|---|
|
||||
| `--mode <mode>` | Filter checks to a specific mode |
|
||||
| `--strict` | Treat warnings as failures |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
|
||||
**Checks:**
|
||||
|
||||
@@ -210,6 +214,31 @@ apophis migrate --dry-run
|
||||
apophis migrate --write
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### CI workflow with machine output
|
||||
```bash
|
||||
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
|
||||
```
|
||||
|
||||
### Monorepo workspace verification
|
||||
```bash
|
||||
apophis verify --workspace --profile quick
|
||||
apophis doctor --workspace
|
||||
```
|
||||
|
||||
### Replay a failure
|
||||
```bash
|
||||
apophis replay --artifact reports/apophis/failure-*.json
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `--changed` requires a git repository
|
||||
- `migrate` defaults to `--dry-run` (safe by default)
|
||||
- `--workspace` is only supported by `verify` and `doctor` commands
|
||||
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
@@ -40,7 +41,7 @@ fastify.post('/users', {
|
||||
}
|
||||
}
|
||||
}, async (req, reply) => {
|
||||
const id = `usr-${Date.now()}`
|
||||
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||
const user = { id, email: req.body.email, name: req.body.name }
|
||||
users.set(id, user)
|
||||
reply.status(201)
|
||||
|
||||
+13
-105
@@ -30,6 +30,8 @@ This creates:
|
||||
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -40,27 +42,20 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
```
|
||||
|
||||
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
|
||||
|
||||
## Step 4: Run Verify
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
APOPHIS will:
|
||||
|
||||
1. Discover routes from your Fastify app
|
||||
2. Filter to `POST /users`
|
||||
3. Generate test data from the schema
|
||||
4. Execute the route
|
||||
5. Check the behavioral contract
|
||||
6. Print pass/fail, seed, and replay command
|
||||
|
||||
## Example Failure
|
||||
|
||||
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
||||
@@ -100,111 +95,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
|
||||
## Next Steps
|
||||
|
||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
||||
- Run all routes: `apophis verify --profile quick`
|
||||
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
||||
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
|
||||
- Requires a git repository.
|
||||
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
||||
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
|
||||
|
||||
## Config Reference
|
||||
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
For the full configuration reference, see [CLI Reference](cli.md).
|
||||
|
||||
## Monorepo Workspaces
|
||||
|
||||
APOPHIS supports workspace-wide operations with the `--workspace` flag.
|
||||
|
||||
### Root package.json scripts
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"apophis:verify": "apophis verify --workspace --profile quick",
|
||||
"apophis:doctor": "apophis doctor --workspace",
|
||||
"apophis:qualify": "apophis qualify --workspace --profile ci"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Workspace fan-out
|
||||
|
||||
Run verify across all packages:
|
||||
Use `--workspace` to run verify or doctor across all packages:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output is package-attributed:
|
||||
|
||||
```json
|
||||
{
|
||||
"exitCode": 0,
|
||||
"runs": [
|
||||
{
|
||||
"package": "api",
|
||||
"cwd": "/repo/packages/api",
|
||||
"artifact": { ... }
|
||||
},
|
||||
{
|
||||
"package": "web",
|
||||
"cwd": "/repo/packages/web",
|
||||
"artifact": { ... }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Supported commands
|
||||
|
||||
- `apophis verify --workspace`
|
||||
- `apophis doctor --workspace`
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Success |
|
||||
| 1 | Behavioral / qualification failure |
|
||||
| 2 | Usage, config, or environment safety violation |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
See [CLI Reference](cli.md) for workspace output format and exit codes.
|
||||
|
||||
@@ -18,10 +18,10 @@ Use `apophis init` with a preset:
|
||||
|
||||
| Preset | Use Case |
|
||||
|---|---|
|
||||
| `safe-ci` | General CI-safe setup |
|
||||
| `llm-safe` | Ultra-minimal for LLM-generated code |
|
||||
| `platform-observe` | Observe-mode policy and runtime drift reporting |
|
||||
| `protocol-lab` | Multi-step flows and stateful testing |
|
||||
| `safe-ci` | Minimal CI-safe preset (default) |
|
||||
| `llm-safe` | Minimal preset for LLM-generated codebases |
|
||||
| `platform-observe` | Production-ready with observe mode |
|
||||
| `protocol-lab` | Multi-step flow and stateful testing |
|
||||
|
||||
```bash
|
||||
apophis init --preset llm-safe
|
||||
@@ -108,6 +108,8 @@ export default {
|
||||
### Route Template with Behavioral Contract
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -134,7 +136,7 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
|
||||
+26
-3
@@ -65,14 +65,16 @@ profiles: {
|
||||
}
|
||||
```
|
||||
|
||||
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema.
|
||||
The `platform-observe` preset enables sampling at the preset level.
|
||||
|
||||
## Staging vs Production
|
||||
|
||||
| Environment | Blocking | Sampling | Sink Required |
|
||||
|---|---|---|---|
|
||||
| Staging | No (default) | 10% | Yes |
|
||||
| Production | No (default) | 1% | Yes |
|
||||
| Staging | No (default) | 100% | Yes |
|
||||
| Production | No (default) | 100% | Yes |
|
||||
|
||||
Default is 1.0 (100%). Configure lower rates for production explicitly.
|
||||
|
||||
## `--check-config` Flag
|
||||
|
||||
@@ -138,3 +140,24 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Sink Endpoint Configuration
|
||||
|
||||
Configure the reporting sink endpoint in your observe config:
|
||||
|
||||
```javascript
|
||||
observe: {
|
||||
sink: {
|
||||
endpoint: 'http://collector.internal:4318'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Workspace Support
|
||||
|
||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages.
|
||||
|
||||
## Mode Mismatch
|
||||
|
||||
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
|
||||
```
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# APOPHIS Protocol Extensions Specification
|
||||
|
||||
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature
|
||||
## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
|
||||
|
||||
## 1. Overview
|
||||
|
||||
@@ -10,7 +10,7 @@ Arbiter maintains 58 protocol conformance test files covering 138 behaviors acro
|
||||
|
||||
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
||||
|
||||
**Shipped in v2.x:**
|
||||
**Shipped in v2.0.0:**
|
||||
|
||||
- `contract({ variants })` for multi-header/media negotiation execution.
|
||||
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
|
||||
@@ -166,12 +166,15 @@ jwtExtension({
|
||||
The JWT extension maintains state across a test run:
|
||||
|
||||
```javascript
|
||||
interface JwtExtensionState {
|
||||
/** Track seen JTIs for replay detection */
|
||||
seenJtis: Set<string>
|
||||
/** Cached decoded JWTs */
|
||||
decodedCache: Map<string, DecodedJwt>
|
||||
}
|
||||
/**
|
||||
* JWT extension state across a test run.
|
||||
* @property {Set<string>} seenJtis - Track seen JTIs for replay detection
|
||||
* @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
|
||||
*/
|
||||
const jwtExtensionState = {
|
||||
seenJtis: new Set(),
|
||||
decodedCache: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 3.5 Example Contracts
|
||||
@@ -234,16 +237,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
|
||||
### 4.4 Implementation
|
||||
|
||||
```javascript
|
||||
interface TimeControl {
|
||||
/** Advance simulated time by milliseconds */
|
||||
advance(ms: number): void
|
||||
/** Set simulated time to specific timestamp */
|
||||
set(isoString: string): void
|
||||
/** Get current simulated time */
|
||||
now(): number
|
||||
/** Reset to real time */
|
||||
reset(): void
|
||||
}
|
||||
/**
|
||||
* Time control for deterministic testing.
|
||||
* @property {function(number): void} advance - Advance simulated time by milliseconds
|
||||
* @property {function(string): void} set - Set simulated time to specific ISO timestamp
|
||||
* @property {function(): number} now - Get current simulated time
|
||||
* @property {function(): void} reset - Reset to real time
|
||||
*/
|
||||
const timeControl = {
|
||||
advance(ms) { /* ... */ },
|
||||
set(isoString) { /* ... */ },
|
||||
now() { return Date.now(); },
|
||||
reset() { /* ... */ }
|
||||
};
|
||||
```
|
||||
|
||||
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
|
||||
@@ -288,11 +294,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
|
||||
Extension state tracks tokens across requests:
|
||||
|
||||
```javascript
|
||||
interface StatefulExtensionState {
|
||||
seenTokens: Set<string>
|
||||
consumedTokens: Set<string>
|
||||
categoryHistory: Map<string, EvalContext> // category -> last context
|
||||
}
|
||||
/**
|
||||
* Stateful extension state tracking tokens across requests.
|
||||
* @property {Set<string>} seenTokens - Tokens observed in previous requests
|
||||
* @property {Set<string>} consumedTokens - Tokens that have been consumed
|
||||
* @property {Map<string, EvalContext>} categoryHistory - category -> last context
|
||||
*/
|
||||
const statefulExtensionState = {
|
||||
seenTokens: new Set(),
|
||||
consumedTokens: new Set(),
|
||||
categoryHistory: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 5.4 Example Contracts
|
||||
@@ -522,14 +534,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
|
||||
## 14. Implementation Plan
|
||||
|
||||
### Phase 1: JWT + Time Control (P0)
|
||||
**Target**: v1.3.0
|
||||
### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/jwt.ts` — JWT extension implementation
|
||||
- `src/extensions/time.ts` — Time control extension
|
||||
- `src/extensions/stateful.ts` — Stateful predicates extension
|
||||
- `src/test/jwt-extension.test.ts` — JWT tests
|
||||
- `src/test/time-extension.test.ts` — Time control tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
- `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
|
||||
|
||||
**Tests**:
|
||||
- Decode Base64URL claims without verification
|
||||
@@ -539,27 +551,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
- `now()` predicate with mocked time
|
||||
- `apophis.time.advance()` in stateful tests
|
||||
|
||||
### Phase 2: X.509 + SPIFFE (P1)
|
||||
**Target**: v1.3.1
|
||||
### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/x509.ts` — X.509 extension
|
||||
- `src/extensions/spiffe.ts` — SPIFFE extension
|
||||
- `src/test/x509-extension.test.ts` — X.509 tests
|
||||
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 3: Token Hash + HTTP Signature (P2)
|
||||
**Target**: v1.3.2
|
||||
### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/token-hash.ts` — Token hash extension
|
||||
- `src/extensions/http-signature.ts` — HTTP signature extension
|
||||
- `src/test/token-hash-extension.test.ts` — Token hash tests
|
||||
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 4: Request Context (P2)
|
||||
**Target**: v1.3.3
|
||||
### Phase 4: Request Context (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/request-context.ts` — Request context predicates
|
||||
- `src/test/request-context-extension.test.ts` — Request context tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
---
|
||||
|
||||
|
||||
+33
-33
@@ -58,7 +58,11 @@ Stateful tests generate sequences of operations and track resources:
|
||||
3. **Observer**: Read resources (GET)
|
||||
4. **Destructor**: Remove resources (DELETE)
|
||||
|
||||
APOPHIS automatically tracks created resources and cleans them up after testing.
|
||||
APOPHIS tracks created resources and runs cleanup after test completion.
|
||||
|
||||
## Route Transparency
|
||||
|
||||
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
|
||||
|
||||
## Chaos and Adversity
|
||||
|
||||
@@ -67,7 +71,9 @@ Chaos testing injects controlled failures:
|
||||
- **Delay**: Slow responses
|
||||
- **Error**: Return error status codes
|
||||
- **Dropout**: Connection failures
|
||||
- **Corruption**: Malformed response bodies
|
||||
- **Truncate**: Truncated response bodies
|
||||
- **Malformed**: Invalid JSON or content-type
|
||||
- **Field-corrupt**: Random field mutation in response objects
|
||||
|
||||
Configure chaos in your preset:
|
||||
|
||||
@@ -84,36 +90,6 @@ presets: {
|
||||
}
|
||||
```
|
||||
|
||||
## Profile Examples
|
||||
|
||||
### oauth-nightly
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'oauth-nightly': {
|
||||
name: 'oauth-nightly',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### lifecycle-deep
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'lifecycle-deep': {
|
||||
name: 'lifecycle-deep',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Non-Prod Boundaries
|
||||
|
||||
Qualify mode is gated away from production by default:
|
||||
@@ -122,7 +98,7 @@ Qualify mode is gated away from production by default:
|
||||
|---|---|---|---|
|
||||
| local | enabled | enabled | enabled |
|
||||
| test/CI | enabled | enabled | enabled |
|
||||
| staging | enabled with allowlist | synthetic-only | canary-only |
|
||||
| staging | enabled with allowlist | enabled | blocked on protected routes |
|
||||
| production | disabled by default | disabled by default | disabled by default |
|
||||
|
||||
## Machine Output for CI
|
||||
@@ -224,3 +200,27 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Gate Execution Counts
|
||||
|
||||
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
|
||||
|
||||
## Zero-Execution Guardrail
|
||||
|
||||
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
|
||||
|
||||
## `--workspace` Flag
|
||||
|
||||
Run qualify across all packages in a monorepo workspace:
|
||||
|
||||
```bash
|
||||
apophis qualify --workspace --profile oauth-nightly
|
||||
```
|
||||
|
||||
## `--generation-profile` Flag
|
||||
|
||||
Control test data generation depth independently from the qualification profile:
|
||||
|
||||
```bash
|
||||
apophis qualify --profile oauth-nightly --generation-profile quick
|
||||
```
|
||||
|
||||
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
|
||||
**Symptoms**
|
||||
- `Unexpected token` in formula output
|
||||
- `Unterminated string` in x-ensures clause
|
||||
- `Unterminated string literal` in x-ensures clause
|
||||
- `Missing this` in operation call
|
||||
|
||||
**Resolution**
|
||||
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
**Symptoms**
|
||||
- `Plugin decorator already added`
|
||||
- `Duplicate route registration`
|
||||
- `No behavioral contracts found`
|
||||
- `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
|
||||
|
||||
**Resolution**
|
||||
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
|
||||
2. Check for multiple imports or plugin registrations in test vs production entry points.
|
||||
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas.
|
||||
3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
|
||||
4. Run `apophis doctor` to verify route discovery matches expectations.
|
||||
|
||||
**Prevention**
|
||||
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
|
||||
|
||||
```bash
|
||||
# Inspect the artifact
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
|
||||
|
||||
# Replay the exact failure
|
||||
apophis replay --artifact reports/apophis/verify-<timestamp>.json
|
||||
apophis replay --artifact reports/apophis/failure-<timestamp>.json
|
||||
|
||||
# Filter by error category
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+27
-35
@@ -2,16 +2,6 @@
|
||||
|
||||
Deterministic contract verification for CI and local development.
|
||||
|
||||
## What Verify Does
|
||||
|
||||
`apophis verify` runs behavioral contracts against your Fastify routes:
|
||||
|
||||
1. Discovers routes from your Fastify app
|
||||
2. Filters routes by profile config and CLI flags
|
||||
3. Generates test data from JSON Schema
|
||||
4. Executes routes and checks `x-ensures` contracts
|
||||
5. Reports pass/fail with deterministic seed and replay command
|
||||
|
||||
## When to Use It
|
||||
|
||||
- **Local development**: Quick feedback on behavioral changes
|
||||
@@ -79,6 +69,8 @@ apophis verify --routes "POST /users/*"
|
||||
apophis verify --profile quick
|
||||
```
|
||||
|
||||
`*` and `?` wildcards are supported in `--routes`.
|
||||
|
||||
## `--changed` Flag
|
||||
|
||||
Run only routes modified in the current git branch:
|
||||
@@ -126,6 +118,8 @@ Next
|
||||
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||
```
|
||||
|
||||
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
|
||||
|
||||
## Machine Output for CI
|
||||
|
||||
Use concise formats to reduce log volume in large verify runs:
|
||||
@@ -137,6 +131,7 @@ Use concise formats to reduce log volume in large verify runs:
|
||||
|
||||
```bash
|
||||
# Extract only failed routes from full ndjson
|
||||
# Note: route.failed events are only emitted for failures, not passed routes
|
||||
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
|
||||
|
||||
# Write artifact to disk and parse the file instead of stdout
|
||||
@@ -149,7 +144,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
|---|---|
|
||||
| 0 | All contracts passed |
|
||||
| 1 | One or more behavioral contracts failed |
|
||||
| 2 | Config error or no routes matched |
|
||||
| 2 | Config error, no routes matched, no contracts found, or not a git repo |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
|
||||
@@ -158,42 +153,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
timeout: 5000
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
For the full config schema, see [CLI Reference](cli.md).
|
||||
|
||||
## Workspace Support
|
||||
|
||||
Run verify across all packages in a monorepo workspace:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output includes per-package pass/fail summaries. Fails if any package fails.
|
||||
|
||||
## `--generation-profile` Flag
|
||||
|
||||
Control test data generation depth independently from the verification profile:
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --generation-profile quick
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user