docs: final cleanup and accuracy pass before public push
- Fix const inference bug: wrap inferred contracts with status-code guards - Add integration test for status-guarded contract inference - Tighten and deduplicate docs across verify, qualify, getting-started, cli - Fix broken cross-references and TypeScript→JavaScript conversions - Fix factual errors: license, Date.now(), sampling defaults, cache env - Add missing features: --workspace, --generation-profile, json-summary formats - Move stale extension docs (AUTH-RATE-LIMIT-REVISED, HTTP-EXTENSIONS) to attic - Update PLUGIN_CONTRACTS_SPEC status to Implemented - Build: clean | Tests: 849 pass, 0 fail
This commit is contained in:
@@ -12,6 +12,8 @@ apophis init --preset safe-ci
|
|||||||
apophis verify --profile quick --routes "POST /users"
|
apophis verify --profile quick --routes "POST /users"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
|
||||||
|
|
||||||
## Cross-Route Failure Example
|
## Cross-Route Failure Example
|
||||||
|
|
||||||
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
|
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
|
||||||
@@ -19,6 +21,8 @@ Add one behavioral contract next to a route schema. APOPHIS can verify cross-rou
|
|||||||
**Route:**
|
**Route:**
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
import crypto from 'crypto';
|
||||||
|
|
||||||
app.post('/users', {
|
app.post('/users', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'constructor',
|
'x-category': 'constructor',
|
||||||
@@ -29,7 +33,7 @@ app.post('/users', {
|
|||||||
}
|
}
|
||||||
}, async (request, reply) => {
|
}, async (request, reply) => {
|
||||||
const { name } = request.body;
|
const { name } = request.body;
|
||||||
const id = `usr-${Date.now()}`;
|
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||||
reply.status(201);
|
reply.status(201);
|
||||||
return { id, name };
|
return { id, name };
|
||||||
});
|
});
|
||||||
@@ -47,7 +51,7 @@ Expected
|
|||||||
response_code(GET /users/{response_body(this).id}) == 200
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
|
|
||||||
Observed
|
Observed
|
||||||
GET /users/usr-123 returned 404
|
GET /users/usr-7d865e returned 404
|
||||||
|
|
||||||
Why this matters
|
Why this matters
|
||||||
The resource created by POST /users is not retrievable.
|
The resource created by POST /users is not retrievable.
|
||||||
@@ -80,6 +84,9 @@ apophis init --preset safe-ci
|
|||||||
|
|
||||||
# 3. Verify
|
# 3. Verify
|
||||||
apophis verify --profile quick --routes "POST /users"
|
apophis verify --profile quick --routes "POST /users"
|
||||||
|
|
||||||
|
# 4. Doctor
|
||||||
|
apophis doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
||||||
@@ -87,9 +94,12 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
|||||||
## Trust and Safety
|
## Trust and Safety
|
||||||
|
|
||||||
- **Deterministic replay**: Every failure includes a seed and a one-command replay.
|
- **Deterministic replay**: Every failure includes a seed and a one-command replay.
|
||||||
|
- **Generation profile aliases**: Control test budget with `--generation-profile quick|standard|deep`.
|
||||||
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
|
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
|
||||||
|
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
|
||||||
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
|
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
|
||||||
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
|
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
|
||||||
|
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
|
||||||
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
|
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
|
||||||
|
|
||||||
## LLM-Safe
|
## LLM-Safe
|
||||||
@@ -116,4 +126,4 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
|||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
ISC
|
MIT
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# APOPHIS Plugin Contract System Specification
|
# APOPHIS Plugin Contract System Specification
|
||||||
|
|
||||||
## Status: Active design; target version to be assigned
|
## Status: Implemented
|
||||||
|
|
||||||
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
|
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
|
||||||
|
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
|||||||
## Operator Resources
|
## Operator Resources
|
||||||
|
|
||||||
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
|
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
|
||||||
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout
|
- [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
|
||||||
|
|
||||||
## CTAs
|
## CTAs
|
||||||
|
|
||||||
|
|||||||
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
|
|||||||
|
|
||||||
| Environment | Cache | Reason |
|
| Environment | Cache | Reason |
|
||||||
|-------------|-------|--------|
|
|-------------|-------|--------|
|
||||||
| `production` | Disabled | No file I/O, no cache hits needed |
|
| `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||||
| `test` | Disabled | Tests should be deterministic, no cache pollution |
|
| `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||||
| `development` | Enabled | Speeds up iterative testing |
|
| `development` | Enabled by default | Speeds up iterative testing |
|
||||||
| default | Enabled | Backward compatible |
|
| default | Enabled by default | Backward compatible |
|
||||||
|
|
||||||
## Cache Invalidation
|
## Cache Invalidation
|
||||||
|
|
||||||
|
|||||||
+19
-19
@@ -4,7 +4,7 @@ Inject controlled failures into contract tests to validate resilience guarantees
|
|||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
```typescript
|
```javascript
|
||||||
const result = await fastify.apophis.contract({
|
const result = await fastify.apophis.contract({
|
||||||
depth: 'standard',
|
depth: 'standard',
|
||||||
chaos: {
|
chaos: {
|
||||||
@@ -14,7 +14,7 @@ const result = await fastify.apophis.contract({
|
|||||||
dropout: { probability: 1 },
|
dropout: { probability: 1 },
|
||||||
corruption: { probability: 1 },
|
corruption: { probability: 1 },
|
||||||
},
|
},
|
||||||
})
|
});
|
||||||
```
|
```
|
||||||
|
|
||||||
## Event Types
|
## Event Types
|
||||||
@@ -52,35 +52,35 @@ Mutates response bodies. Tests parsing robustness:
|
|||||||
response_body(this).id != null
|
response_body(this).id != null
|
||||||
```
|
```
|
||||||
|
|
||||||
## Content-Type Aware Corruption
|
## Corruption Strategies
|
||||||
|
|
||||||
Built-in strategies for common formats:
|
Built-in strategies are content-type agnostic:
|
||||||
|
|
||||||
| Content-Type | Strategy | Effect |
|
| Strategy | Effect |
|
||||||
|-------------|----------|--------|
|
|----------|--------|
|
||||||
| `application/json` | Truncate or null field | Removes fields or sets random field to null |
|
| `truncate` | Cuts response body short |
|
||||||
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk |
|
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
|
||||||
| `text/event-stream` | Event corrupt | Adds malformed SSE line |
|
| `field-corrupt` | Replaces a random field value with corrupted data |
|
||||||
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
|
|
||||||
| `text/plain` | Truncate | Cuts string in half |
|
Extension strategies can add content-type-specific behavior if needed.
|
||||||
|
|
||||||
## Custom Corruption via Extensions
|
## Custom Corruption via Extensions
|
||||||
|
|
||||||
```typescript
|
```javascript
|
||||||
const myExtension = {
|
const myExtension = {
|
||||||
name: 'custom-corrupt',
|
name: 'custom-corrupt',
|
||||||
corruptionStrategies: {
|
corruptionStrategies: {
|
||||||
'application/vnd.api+json': (data) => ({
|
'application/vnd.api+json': (data) => ({
|
||||||
...data as object,
|
...data,
|
||||||
corrupted: true,
|
corrupted: true,
|
||||||
}),
|
}),
|
||||||
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
||||||
},
|
},
|
||||||
}
|
};
|
||||||
|
|
||||||
await fastify.register(apophis, {
|
await fastify.register(apophis, {
|
||||||
extensions: [myExtension],
|
extensions: [myExtension],
|
||||||
})
|
});
|
||||||
```
|
```
|
||||||
|
|
||||||
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
||||||
@@ -90,7 +90,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
|
|||||||
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
||||||
|
|
||||||
```
|
```
|
||||||
Error: Chaos mode is only available in test environment.
|
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
|
||||||
```
|
```
|
||||||
|
|
||||||
## Interpreting Results
|
## Interpreting Results
|
||||||
@@ -123,7 +123,7 @@ Failed tests include chaos events in diagnostics:
|
|||||||
|
|
||||||
## Example: Testing Retry Logic
|
## Example: Testing Retry Logic
|
||||||
|
|
||||||
```typescript
|
```javascript
|
||||||
fastify.get('/data', {
|
fastify.get('/data', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
@@ -131,7 +131,7 @@ fastify.get('/data', {
|
|||||||
'redirect_count(this) <= 3',
|
'redirect_count(this) <= 3',
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
}, handler)
|
}, handler);
|
||||||
|
|
||||||
// Test
|
// Test
|
||||||
const result = await fastify.apophis.contract({
|
const result = await fastify.apophis.contract({
|
||||||
@@ -139,5 +139,5 @@ const result = await fastify.apophis.contract({
|
|||||||
probability: 0.2,
|
probability: 0.2,
|
||||||
error: { probability: 1, statusCode: 503 },
|
error: { probability: 1, statusCode: 503 },
|
||||||
},
|
},
|
||||||
})
|
});
|
||||||
```
|
```
|
||||||
|
|||||||
+33
-4
@@ -10,15 +10,17 @@ Every command accepts these flags:
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--config <path>` | Config file path | Auto-detect |
|
| `--config <path>` | Config file path | Auto-detect |
|
||||||
| `--profile <name>` | Profile name from config | First profile |
|
| `--profile <name>` | Profile name from config | First profile |
|
||||||
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived |
|
| `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
|
||||||
| `--cwd <path>` | Working directory override | `process.cwd()` |
|
| `--cwd <path>` | Working directory override | `process.cwd()` |
|
||||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
|
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
|
||||||
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
|
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
|
||||||
| `--quiet` | Suppress non-error output | false |
|
| `--quiet` | Suppress non-error output | false |
|
||||||
| `--verbose` | Enable verbose logging | false |
|
| `--verbose` | Enable verbose logging | false |
|
||||||
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` |
|
| `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
|
||||||
| `--workspace` | Run supported commands across workspace packages | false |
|
| `--workspace` | Run supported commands across workspace packages | false |
|
||||||
|
|
||||||
|
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
### `apophis init`
|
### `apophis init`
|
||||||
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
|
|||||||
|
|
||||||
| Flag | Description |
|
| Flag | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
| `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||||
| `--force` | Overwrite existing files |
|
| `-f, --force` | Overwrite existing files |
|
||||||
| `--noninteractive` | Skip all prompts, require explicit flags |
|
| `--noninteractive` | Skip all prompts, require explicit flags |
|
||||||
|
|
||||||
**Examples:**
|
**Examples:**
|
||||||
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
|
|||||||
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
||||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||||
| `--changed` | Filter to git-modified routes only |
|
| `--changed` | Filter to git-modified routes only |
|
||||||
|
| `--workspace` | Run across all workspace packages |
|
||||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
||||||
|
|
||||||
**Examples:**
|
**Examples:**
|
||||||
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `--mode <mode>` | Filter checks to a specific mode |
|
| `--mode <mode>` | Filter checks to a specific mode |
|
||||||
| `--strict` | Treat warnings as failures |
|
| `--strict` | Treat warnings as failures |
|
||||||
|
| `--workspace` | Run across all workspace packages |
|
||||||
|
|
||||||
**Checks:**
|
**Checks:**
|
||||||
|
|
||||||
@@ -210,6 +214,31 @@ apophis migrate --dry-run
|
|||||||
apophis migrate --write
|
apophis migrate --write
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Common Tasks
|
||||||
|
|
||||||
|
### CI workflow with machine output
|
||||||
|
```bash
|
||||||
|
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monorepo workspace verification
|
||||||
|
```bash
|
||||||
|
apophis verify --workspace --profile quick
|
||||||
|
apophis doctor --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
### Replay a failure
|
||||||
|
```bash
|
||||||
|
apophis replay --artifact reports/apophis/failure-*.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- `--changed` requires a git repository
|
||||||
|
- `migrate` defaults to `--dry-run` (safe by default)
|
||||||
|
- `--workspace` is only supported by `verify` and `doctor` commands
|
||||||
|
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
||||||
|
|
||||||
## Exit Codes
|
## Exit Codes
|
||||||
|
|
||||||
| Code | Meaning |
|
| Code | Meaning |
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import Fastify from 'fastify'
|
import Fastify from 'fastify'
|
||||||
import apophisPlugin from 'apophis-fastify'
|
import apophisPlugin from 'apophis-fastify'
|
||||||
|
import crypto from 'crypto'
|
||||||
|
|
||||||
const fastify = Fastify()
|
const fastify = Fastify()
|
||||||
|
|
||||||
@@ -40,7 +41,7 @@ fastify.post('/users', {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}, async (req, reply) => {
|
}, async (req, reply) => {
|
||||||
const id = `usr-${Date.now()}`
|
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||||
const user = { id, email: req.body.email, name: req.body.name }
|
const user = { id, email: req.body.email, name: req.body.name }
|
||||||
users.set(id, user)
|
users.set(id, user)
|
||||||
reply.status(201)
|
reply.status(201)
|
||||||
|
|||||||
+13
-105
@@ -30,6 +30,8 @@ This creates:
|
|||||||
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
import crypto from 'crypto';
|
||||||
|
|
||||||
app.post('/users', {
|
app.post('/users', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'constructor',
|
'x-category': 'constructor',
|
||||||
@@ -40,27 +42,20 @@ app.post('/users', {
|
|||||||
}
|
}
|
||||||
}, async (request, reply) => {
|
}, async (request, reply) => {
|
||||||
const { name } = request.body;
|
const { name } = request.body;
|
||||||
const id = `usr-${Date.now()}`;
|
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||||
reply.status(201);
|
reply.status(201);
|
||||||
return { id, name };
|
return { id, name };
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
|
||||||
|
|
||||||
## Step 4: Run Verify
|
## Step 4: Run Verify
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
apophis verify --profile quick --routes "POST /users"
|
apophis verify --profile quick --routes "POST /users"
|
||||||
```
|
```
|
||||||
|
|
||||||
APOPHIS will:
|
|
||||||
|
|
||||||
1. Discover routes from your Fastify app
|
|
||||||
2. Filter to `POST /users`
|
|
||||||
3. Generate test data from the schema
|
|
||||||
4. Execute the route
|
|
||||||
5. Check the behavioral contract
|
|
||||||
6. Print pass/fail, seed, and replay command
|
|
||||||
|
|
||||||
## Example Failure
|
## Example Failure
|
||||||
|
|
||||||
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
||||||
@@ -100,111 +95,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
|
|||||||
## Next Steps
|
## Next Steps
|
||||||
|
|
||||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||||
|
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
||||||
- Run all routes: `apophis verify --profile quick`
|
- Run all routes: `apophis verify --profile quick`
|
||||||
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
||||||
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
|
- Requires a git repository.
|
||||||
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
|
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
||||||
|
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
||||||
|
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
|
||||||
|
|
||||||
## Config Reference
|
## Config Reference
|
||||||
|
|
||||||
```javascript
|
For the full configuration reference, see [CLI Reference](cli.md).
|
||||||
// apophis.config.js
|
|
||||||
export default {
|
|
||||||
mode: 'verify',
|
|
||||||
profile: 'quick',
|
|
||||||
profiles: {
|
|
||||||
quick: {
|
|
||||||
name: 'quick',
|
|
||||||
mode: 'verify',
|
|
||||||
preset: 'safe-ci',
|
|
||||||
routes: ['POST /users']
|
|
||||||
},
|
|
||||||
ci: {
|
|
||||||
name: 'ci',
|
|
||||||
mode: 'verify',
|
|
||||||
preset: 'safe-ci',
|
|
||||||
routes: []
|
|
||||||
}
|
|
||||||
},
|
|
||||||
presets: {
|
|
||||||
'safe-ci': {
|
|
||||||
name: 'safe-ci',
|
|
||||||
depth: 'quick',
|
|
||||||
timeout: 5000,
|
|
||||||
parallel: false,
|
|
||||||
chaos: false,
|
|
||||||
observe: false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
environments: {
|
|
||||||
local: {
|
|
||||||
name: 'local',
|
|
||||||
allowVerify: true,
|
|
||||||
allowObserve: true,
|
|
||||||
allowQualify: false,
|
|
||||||
allowChaos: false,
|
|
||||||
allowBlocking: true,
|
|
||||||
requireSink: false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monorepo Workspaces
|
## Monorepo Workspaces
|
||||||
|
|
||||||
APOPHIS supports workspace-wide operations with the `--workspace` flag.
|
Use `--workspace` to run verify or doctor across all packages:
|
||||||
|
|
||||||
### Root package.json scripts
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"scripts": {
|
|
||||||
"apophis:verify": "apophis verify --workspace --profile quick",
|
|
||||||
"apophis:doctor": "apophis doctor --workspace",
|
|
||||||
"apophis:qualify": "apophis qualify --workspace --profile ci"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Workspace fan-out
|
|
||||||
|
|
||||||
Run verify across all packages:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
apophis verify --workspace --profile quick --format json
|
apophis verify --workspace --profile quick --format json
|
||||||
```
|
```
|
||||||
|
|
||||||
Output is package-attributed:
|
See [CLI Reference](cli.md) for workspace output format and exit codes.
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"exitCode": 0,
|
|
||||||
"runs": [
|
|
||||||
{
|
|
||||||
"package": "api",
|
|
||||||
"cwd": "/repo/packages/api",
|
|
||||||
"artifact": { ... }
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"package": "web",
|
|
||||||
"cwd": "/repo/packages/web",
|
|
||||||
"artifact": { ... }
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Supported commands
|
|
||||||
|
|
||||||
- `apophis verify --workspace`
|
|
||||||
- `apophis doctor --workspace`
|
|
||||||
|
|
||||||
## Exit Codes
|
|
||||||
|
|
||||||
| Code | Meaning |
|
|
||||||
|---|---|
|
|
||||||
| 0 | Success |
|
|
||||||
| 1 | Behavioral / qualification failure |
|
|
||||||
| 2 | Usage, config, or environment safety violation |
|
|
||||||
| 3 | Internal APOPHIS error |
|
|
||||||
| 130 | Interrupted (SIGINT) |
|
|
||||||
|
|||||||
@@ -18,10 +18,10 @@ Use `apophis init` with a preset:
|
|||||||
|
|
||||||
| Preset | Use Case |
|
| Preset | Use Case |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `safe-ci` | General CI-safe setup |
|
| `safe-ci` | Minimal CI-safe preset (default) |
|
||||||
| `llm-safe` | Ultra-minimal for LLM-generated code |
|
| `llm-safe` | Minimal preset for LLM-generated codebases |
|
||||||
| `platform-observe` | Observe-mode policy and runtime drift reporting |
|
| `platform-observe` | Production-ready with observe mode |
|
||||||
| `protocol-lab` | Multi-step flows and stateful testing |
|
| `protocol-lab` | Multi-step flow and stateful testing |
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
apophis init --preset llm-safe
|
apophis init --preset llm-safe
|
||||||
@@ -108,6 +108,8 @@ export default {
|
|||||||
### Route Template with Behavioral Contract
|
### Route Template with Behavioral Contract
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
import crypto from 'crypto';
|
||||||
|
|
||||||
app.post('/users', {
|
app.post('/users', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'constructor',
|
'x-category': 'constructor',
|
||||||
@@ -134,7 +136,7 @@ app.post('/users', {
|
|||||||
}
|
}
|
||||||
}, async (request, reply) => {
|
}, async (request, reply) => {
|
||||||
const { name } = request.body;
|
const { name } = request.body;
|
||||||
const id = `usr-${Date.now()}`;
|
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||||
reply.status(201);
|
reply.status(201);
|
||||||
return { id, name };
|
return { id, name };
|
||||||
});
|
});
|
||||||
|
|||||||
+26
-3
@@ -65,14 +65,16 @@ profiles: {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema.
|
The `platform-observe` preset enables sampling at the preset level.
|
||||||
|
|
||||||
## Staging vs Production
|
## Staging vs Production
|
||||||
|
|
||||||
| Environment | Blocking | Sampling | Sink Required |
|
| Environment | Blocking | Sampling | Sink Required |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Staging | No (default) | 10% | Yes |
|
| Staging | No (default) | 100% | Yes |
|
||||||
| Production | No (default) | 1% | Yes |
|
| Production | No (default) | 100% | Yes |
|
||||||
|
|
||||||
|
Default is 1.0 (100%). Configure lower rates for production explicitly.
|
||||||
|
|
||||||
## `--check-config` Flag
|
## `--check-config` Flag
|
||||||
|
|
||||||
@@ -138,3 +140,24 @@ export default {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Sink Endpoint Configuration
|
||||||
|
|
||||||
|
Configure the reporting sink endpoint in your observe config:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
observe: {
|
||||||
|
sink: {
|
||||||
|
endpoint: 'http://collector.internal:4318'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workspace Support
|
||||||
|
|
||||||
|
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages.
|
||||||
|
|
||||||
|
## Mode Mismatch
|
||||||
|
|
||||||
|
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
|
||||||
|
```
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# APOPHIS Protocol Extensions Specification
|
# APOPHIS Protocol Extensions Specification
|
||||||
|
|
||||||
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature
|
## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
|
||||||
|
|
||||||
## 1. Overview
|
## 1. Overview
|
||||||
|
|
||||||
@@ -10,7 +10,7 @@ Arbiter maintains 58 protocol conformance test files covering 138 behaviors acro
|
|||||||
|
|
||||||
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
||||||
|
|
||||||
**Shipped in v2.x:**
|
**Shipped in v2.0.0:**
|
||||||
|
|
||||||
- `contract({ variants })` for multi-header/media negotiation execution.
|
- `contract({ variants })` for multi-header/media negotiation execution.
|
||||||
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
|
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
|
||||||
@@ -166,12 +166,15 @@ jwtExtension({
|
|||||||
The JWT extension maintains state across a test run:
|
The JWT extension maintains state across a test run:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
interface JwtExtensionState {
|
/**
|
||||||
/** Track seen JTIs for replay detection */
|
* JWT extension state across a test run.
|
||||||
seenJtis: Set<string>
|
* @property {Set<string>} seenJtis - Track seen JTIs for replay detection
|
||||||
/** Cached decoded JWTs */
|
* @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
|
||||||
decodedCache: Map<string, DecodedJwt>
|
*/
|
||||||
}
|
const jwtExtensionState = {
|
||||||
|
seenJtis: new Set(),
|
||||||
|
decodedCache: new Map()
|
||||||
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3.5 Example Contracts
|
### 3.5 Example Contracts
|
||||||
@@ -234,16 +237,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
|
|||||||
### 4.4 Implementation
|
### 4.4 Implementation
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
interface TimeControl {
|
/**
|
||||||
/** Advance simulated time by milliseconds */
|
* Time control for deterministic testing.
|
||||||
advance(ms: number): void
|
* @property {function(number): void} advance - Advance simulated time by milliseconds
|
||||||
/** Set simulated time to specific timestamp */
|
* @property {function(string): void} set - Set simulated time to specific ISO timestamp
|
||||||
set(isoString: string): void
|
* @property {function(): number} now - Get current simulated time
|
||||||
/** Get current simulated time */
|
* @property {function(): void} reset - Reset to real time
|
||||||
now(): number
|
*/
|
||||||
/** Reset to real time */
|
const timeControl = {
|
||||||
reset(): void
|
advance(ms) { /* ... */ },
|
||||||
}
|
set(isoString) { /* ... */ },
|
||||||
|
now() { return Date.now(); },
|
||||||
|
reset() { /* ... */ }
|
||||||
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
|
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
|
||||||
@@ -288,11 +294,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
|
|||||||
Extension state tracks tokens across requests:
|
Extension state tracks tokens across requests:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
interface StatefulExtensionState {
|
/**
|
||||||
seenTokens: Set<string>
|
* Stateful extension state tracking tokens across requests.
|
||||||
consumedTokens: Set<string>
|
* @property {Set<string>} seenTokens - Tokens observed in previous requests
|
||||||
categoryHistory: Map<string, EvalContext> // category -> last context
|
* @property {Set<string>} consumedTokens - Tokens that have been consumed
|
||||||
}
|
* @property {Map<string, EvalContext>} categoryHistory - category -> last context
|
||||||
|
*/
|
||||||
|
const statefulExtensionState = {
|
||||||
|
seenTokens: new Set(),
|
||||||
|
consumedTokens: new Set(),
|
||||||
|
categoryHistory: new Map()
|
||||||
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5.4 Example Contracts
|
### 5.4 Example Contracts
|
||||||
@@ -522,14 +534,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
|||||||
|
|
||||||
## 14. Implementation Plan
|
## 14. Implementation Plan
|
||||||
|
|
||||||
### Phase 1: JWT + Time Control (P0)
|
### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
|
||||||
**Target**: v1.3.0
|
**Status**: Complete
|
||||||
**Files**:
|
**Files**:
|
||||||
- `src/extensions/jwt.ts` — JWT extension implementation
|
- `src/extensions/jwt.ts` — JWT extension implementation
|
||||||
- `src/extensions/time.ts` — Time control extension
|
- `src/extensions/time.ts` — Time control extension
|
||||||
- `src/extensions/stateful.ts` — Stateful predicates extension
|
- `src/extensions/stateful.ts` — Stateful predicates extension
|
||||||
- `src/test/jwt-extension.test.ts` — JWT tests
|
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||||
- `src/test/time-extension.test.ts` — Time control tests
|
- `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
|
||||||
|
|
||||||
**Tests**:
|
**Tests**:
|
||||||
- Decode Base64URL claims without verification
|
- Decode Base64URL claims without verification
|
||||||
@@ -539,27 +551,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
|||||||
- `now()` predicate with mocked time
|
- `now()` predicate with mocked time
|
||||||
- `apophis.time.advance()` in stateful tests
|
- `apophis.time.advance()` in stateful tests
|
||||||
|
|
||||||
### Phase 2: X.509 + SPIFFE (P1)
|
### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
|
||||||
**Target**: v1.3.1
|
**Status**: Complete
|
||||||
**Files**:
|
**Files**:
|
||||||
- `src/extensions/x509.ts` — X.509 extension
|
- `src/extensions/x509.ts` — X.509 extension
|
||||||
- `src/extensions/spiffe.ts` — SPIFFE extension
|
- `src/extensions/spiffe.ts` — SPIFFE extension
|
||||||
- `src/test/x509-extension.test.ts` — X.509 tests
|
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||||
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
|
|
||||||
|
|
||||||
### Phase 3: Token Hash + HTTP Signature (P2)
|
### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
|
||||||
**Target**: v1.3.2
|
**Status**: Complete
|
||||||
**Files**:
|
**Files**:
|
||||||
- `src/extensions/token-hash.ts` — Token hash extension
|
- `src/extensions/token-hash.ts` — Token hash extension
|
||||||
- `src/extensions/http-signature.ts` — HTTP signature extension
|
- `src/extensions/http-signature.ts` — HTTP signature extension
|
||||||
- `src/test/token-hash-extension.test.ts` — Token hash tests
|
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||||
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
|
|
||||||
|
|
||||||
### Phase 4: Request Context (P2)
|
### Phase 4: Request Context (P2) — Shipped in v2.0.0
|
||||||
**Target**: v1.3.3
|
**Status**: Complete
|
||||||
**Files**:
|
**Files**:
|
||||||
- `src/extensions/request-context.ts` — Request context predicates
|
- `src/extensions/request-context.ts` — Request context predicates
|
||||||
- `src/test/request-context-extension.test.ts` — Request context tests
|
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+33
-33
@@ -58,7 +58,11 @@ Stateful tests generate sequences of operations and track resources:
|
|||||||
3. **Observer**: Read resources (GET)
|
3. **Observer**: Read resources (GET)
|
||||||
4. **Destructor**: Remove resources (DELETE)
|
4. **Destructor**: Remove resources (DELETE)
|
||||||
|
|
||||||
APOPHIS automatically tracks created resources and cleans them up after testing.
|
APOPHIS tracks created resources and runs cleanup after test completion.
|
||||||
|
|
||||||
|
## Route Transparency
|
||||||
|
|
||||||
|
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
|
||||||
|
|
||||||
## Chaos and Adversity
|
## Chaos and Adversity
|
||||||
|
|
||||||
@@ -67,7 +71,9 @@ Chaos testing injects controlled failures:
|
|||||||
- **Delay**: Slow responses
|
- **Delay**: Slow responses
|
||||||
- **Error**: Return error status codes
|
- **Error**: Return error status codes
|
||||||
- **Dropout**: Connection failures
|
- **Dropout**: Connection failures
|
||||||
- **Corruption**: Malformed response bodies
|
- **Truncate**: Truncated response bodies
|
||||||
|
- **Malformed**: Invalid JSON or content-type
|
||||||
|
- **Field-corrupt**: Random field mutation in response objects
|
||||||
|
|
||||||
Configure chaos in your preset:
|
Configure chaos in your preset:
|
||||||
|
|
||||||
@@ -84,36 +90,6 @@ presets: {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Profile Examples
|
|
||||||
|
|
||||||
### oauth-nightly
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
profiles: {
|
|
||||||
'oauth-nightly': {
|
|
||||||
name: 'oauth-nightly',
|
|
||||||
mode: 'qualify',
|
|
||||||
preset: 'protocol-lab',
|
|
||||||
routes: [],
|
|
||||||
seed: 42
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### lifecycle-deep
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
profiles: {
|
|
||||||
'lifecycle-deep': {
|
|
||||||
name: 'lifecycle-deep',
|
|
||||||
mode: 'qualify',
|
|
||||||
preset: 'protocol-lab',
|
|
||||||
routes: [],
|
|
||||||
seed: 42
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Non-Prod Boundaries
|
## Non-Prod Boundaries
|
||||||
|
|
||||||
Qualify mode is gated away from production by default:
|
Qualify mode is gated away from production by default:
|
||||||
@@ -122,7 +98,7 @@ Qualify mode is gated away from production by default:
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| local | enabled | enabled | enabled |
|
| local | enabled | enabled | enabled |
|
||||||
| test/CI | enabled | enabled | enabled |
|
| test/CI | enabled | enabled | enabled |
|
||||||
| staging | enabled with allowlist | synthetic-only | canary-only |
|
| staging | enabled with allowlist | enabled | blocked on protected routes |
|
||||||
| production | disabled by default | disabled by default | disabled by default |
|
| production | disabled by default | disabled by default | disabled by default |
|
||||||
|
|
||||||
## Machine Output for CI
|
## Machine Output for CI
|
||||||
@@ -224,3 +200,27 @@ export default {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Gate Execution Counts
|
||||||
|
|
||||||
|
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
|
||||||
|
|
||||||
|
## Zero-Execution Guardrail
|
||||||
|
|
||||||
|
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
|
||||||
|
|
||||||
|
## `--workspace` Flag
|
||||||
|
|
||||||
|
Run qualify across all packages in a monorepo workspace:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apophis qualify --workspace --profile oauth-nightly
|
||||||
|
```
|
||||||
|
|
||||||
|
## `--generation-profile` Flag
|
||||||
|
|
||||||
|
Control test data generation depth independently from the qualification profile:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apophis qualify --profile oauth-nightly --generation-profile quick
|
||||||
|
```
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
|||||||
|
|
||||||
**Symptoms**
|
**Symptoms**
|
||||||
- `Unexpected token` in formula output
|
- `Unexpected token` in formula output
|
||||||
- `Unterminated string` in x-ensures clause
|
- `Unterminated string literal` in x-ensures clause
|
||||||
- `Missing this` in operation call
|
- `Missing this` in operation call
|
||||||
|
|
||||||
**Resolution**
|
**Resolution**
|
||||||
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
|||||||
**Symptoms**
|
**Symptoms**
|
||||||
- `Plugin decorator already added`
|
- `Plugin decorator already added`
|
||||||
- `Duplicate route registration`
|
- `Duplicate route registration`
|
||||||
- `No behavioral contracts found`
|
- `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
|
||||||
|
|
||||||
**Resolution**
|
**Resolution**
|
||||||
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
|
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
|
||||||
2. Check for multiple imports or plugin registrations in test vs production entry points.
|
2. Check for multiple imports or plugin registrations in test vs production entry points.
|
||||||
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas.
|
3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
|
||||||
4. Run `apophis doctor` to verify route discovery matches expectations.
|
4. Run `apophis doctor` to verify route discovery matches expectations.
|
||||||
|
|
||||||
**Prevention**
|
**Prevention**
|
||||||
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Inspect the artifact
|
# Inspect the artifact
|
||||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]'
|
cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
|
||||||
|
|
||||||
# Replay the exact failure
|
# Replay the exact failure
|
||||||
apophis replay --artifact reports/apophis/verify-<timestamp>.json
|
apophis replay --artifact reports/apophis/failure-<timestamp>.json
|
||||||
|
|
||||||
# Filter by error category
|
# Filter by error category
|
||||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
+27
-35
@@ -2,16 +2,6 @@
|
|||||||
|
|
||||||
Deterministic contract verification for CI and local development.
|
Deterministic contract verification for CI and local development.
|
||||||
|
|
||||||
## What Verify Does
|
|
||||||
|
|
||||||
`apophis verify` runs behavioral contracts against your Fastify routes:
|
|
||||||
|
|
||||||
1. Discovers routes from your Fastify app
|
|
||||||
2. Filters routes by profile config and CLI flags
|
|
||||||
3. Generates test data from JSON Schema
|
|
||||||
4. Executes routes and checks `x-ensures` contracts
|
|
||||||
5. Reports pass/fail with deterministic seed and replay command
|
|
||||||
|
|
||||||
## When to Use It
|
## When to Use It
|
||||||
|
|
||||||
- **Local development**: Quick feedback on behavioral changes
|
- **Local development**: Quick feedback on behavioral changes
|
||||||
@@ -79,6 +69,8 @@ apophis verify --routes "POST /users/*"
|
|||||||
apophis verify --profile quick
|
apophis verify --profile quick
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`*` and `?` wildcards are supported in `--routes`.
|
||||||
|
|
||||||
## `--changed` Flag
|
## `--changed` Flag
|
||||||
|
|
||||||
Run only routes modified in the current git branch:
|
Run only routes modified in the current git branch:
|
||||||
@@ -126,6 +118,8 @@ Next
|
|||||||
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
|
||||||
|
|
||||||
## Machine Output for CI
|
## Machine Output for CI
|
||||||
|
|
||||||
Use concise formats to reduce log volume in large verify runs:
|
Use concise formats to reduce log volume in large verify runs:
|
||||||
@@ -137,6 +131,7 @@ Use concise formats to reduce log volume in large verify runs:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Extract only failed routes from full ndjson
|
# Extract only failed routes from full ndjson
|
||||||
|
# Note: route.failed events are only emitted for failures, not passed routes
|
||||||
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
|
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
|
||||||
|
|
||||||
# Write artifact to disk and parse the file instead of stdout
|
# Write artifact to disk and parse the file instead of stdout
|
||||||
@@ -149,7 +144,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
|||||||
|---|---|
|
|---|---|
|
||||||
| 0 | All contracts passed |
|
| 0 | All contracts passed |
|
||||||
| 1 | One or more behavioral contracts failed |
|
| 1 | One or more behavioral contracts failed |
|
||||||
| 2 | Config error or no routes matched |
|
| 2 | Config error, no routes matched, no contracts found, or not a git repo |
|
||||||
| 3 | Internal APOPHIS error |
|
| 3 | Internal APOPHIS error |
|
||||||
| 130 | Interrupted (SIGINT) |
|
| 130 | Interrupted (SIGINT) |
|
||||||
|
|
||||||
@@ -158,42 +153,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
|||||||
```javascript
|
```javascript
|
||||||
// apophis.config.js
|
// apophis.config.js
|
||||||
export default {
|
export default {
|
||||||
mode: 'verify',
|
|
||||||
profile: 'quick',
|
profile: 'quick',
|
||||||
profiles: {
|
profiles: {
|
||||||
quick: {
|
quick: {
|
||||||
name: 'quick',
|
|
||||||
mode: 'verify',
|
mode: 'verify',
|
||||||
preset: 'safe-ci',
|
preset: 'safe-ci',
|
||||||
routes: ['POST /users']
|
routes: ['POST /users']
|
||||||
},
|
|
||||||
ci: {
|
|
||||||
name: 'ci',
|
|
||||||
mode: 'verify',
|
|
||||||
preset: 'safe-ci',
|
|
||||||
routes: []
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
presets: {
|
presets: {
|
||||||
'safe-ci': {
|
'safe-ci': {
|
||||||
name: 'safe-ci',
|
|
||||||
depth: 'quick',
|
depth: 'quick',
|
||||||
timeout: 5000,
|
timeout: 5000
|
||||||
parallel: false,
|
|
||||||
chaos: false,
|
|
||||||
observe: false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
environments: {
|
|
||||||
local: {
|
|
||||||
name: 'local',
|
|
||||||
allowVerify: true,
|
|
||||||
allowObserve: true,
|
|
||||||
allowQualify: false,
|
|
||||||
allowChaos: false,
|
|
||||||
allowBlocking: true,
|
|
||||||
requireSink: false
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For the full config schema, see [CLI Reference](cli.md).
|
||||||
|
|
||||||
|
## Workspace Support
|
||||||
|
|
||||||
|
Run verify across all packages in a monorepo workspace:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apophis verify --workspace --profile quick --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Output includes per-package pass/fail summaries. Fails if any package fails.
|
||||||
|
|
||||||
|
## `--generation-profile` Flag
|
||||||
|
|
||||||
|
Control test data generation depth independently from the verification profile:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apophis verify --profile quick --generation-profile quick
|
||||||
|
```
|
||||||
|
|||||||
+2170
File diff suppressed because it is too large
Load Diff
@@ -7,13 +7,13 @@
|
|||||||
*
|
*
|
||||||
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
|
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
|
||||||
*
|
*
|
||||||
* Supported inference:
|
* Supported inference (all wrapped with status-code guard):
|
||||||
* - required fields → response_body(this).field != null
|
* - required fields → response_code(this) == N => response_body(this).field != null
|
||||||
* - minimum (number/integer) → response_body(this).field >= N
|
* - minimum (number/integer) → response_code(this) == N => response_body(this).field >= N
|
||||||
* - maximum (number/integer) → response_body(this).field <= N
|
* - maximum (number/integer) → response_code(this) == N => response_body(this).field <= N
|
||||||
* - pattern (string) → response_body(this).field matches "..."
|
* - pattern (string) → response_code(this) == N => response_body(this).field matches "..."
|
||||||
* - const → response_body(this).field == value
|
* - const → response_code(this) == N => response_body(this).field == value
|
||||||
* - enum (small) → response_body(this).field == "a" || response_body(this).field == "b"
|
* - enum (small) → response_code(this) == N => response_body(this).field == "a" || ...
|
||||||
*
|
*
|
||||||
* Not inferred (leave to x-ensures for business logic):
|
* Not inferred (leave to x-ensures for business logic):
|
||||||
* - minLength/maxLength
|
* - minLength/maxLength
|
||||||
@@ -188,7 +188,12 @@ export function inferContractsFromRouteSchema(
|
|||||||
const code = parseInt(statusCode, 10)
|
const code = parseInt(statusCode, 10)
|
||||||
if (code >= 200 && code < 300) {
|
if (code >= 200 && code < 300) {
|
||||||
const inferred = inferContractsFromResponseSchema(statusSchema)
|
const inferred = inferContractsFromResponseSchema(statusSchema)
|
||||||
formulas.push(...inferred)
|
// Wrap each inferred contract with a status-code guard so it only
|
||||||
|
// applies when the response actually matches the schema it was
|
||||||
|
// inferred from. Prevents a 200-schema const from failing on a 404.
|
||||||
|
for (const formula of inferred) {
|
||||||
|
formulas.push(`response_code(this) == ${code} => ${formula}`)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -753,3 +753,40 @@ test('integration: route-level x-variants are extracted and executed', async ()
|
|||||||
await fastify.close()
|
await fastify.close()
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('integration: inferred contracts are guarded by status code', async () => {
|
||||||
|
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||||
|
try {
|
||||||
|
await fastify.register(swagger, {})
|
||||||
|
await fastify.register(apophisPlugin, {})
|
||||||
|
fastify.get('/status-guarded', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
response: {
|
||||||
|
200: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { status: { type: 'string', const: 'success' } },
|
||||||
|
required: ['status']
|
||||||
|
},
|
||||||
|
404: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { error: { type: 'string' } },
|
||||||
|
required: ['error']
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} as Record<string, unknown>
|
||||||
|
}, async (request, reply) => {
|
||||||
|
// Return 404 to verify the 200-schema const doesn't fail
|
||||||
|
reply.status(404)
|
||||||
|
return { error: 'not found' }
|
||||||
|
})
|
||||||
|
await fastify.ready()
|
||||||
|
const result = await fastify.apophis.contract({ depth: 'quick' })
|
||||||
|
// Should pass because the inferred const contract is guarded:
|
||||||
|
// response_code(this) == 200 => response_body(this).status == "success"
|
||||||
|
// The 404 response doesn't trigger the antecedent, so the implication holds.
|
||||||
|
assert.strictEqual(result.summary.failed, 0, 'inferred 200-schema const should not fail on 404')
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|||||||
Reference in New Issue
Block a user