Compare commits
3 Commits
v2.0.0
...
bf7376b5ad
| Author | SHA1 | Date | |
|---|---|---|---|
| bf7376b5ad | |||
| 8d7382417d | |||
| 6c39bd0a6c |
@@ -4,6 +4,8 @@ Behavioral confidence for Fastify services.
|
||||
|
||||
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
|
||||
|
||||
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
|
||||
|
||||
Supported Node.js versions: 20.x and 22.x.
|
||||
|
||||
```bash
|
||||
@@ -12,6 +14,8 @@ apophis init --preset safe-ci
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
|
||||
|
||||
## Cross-Route Failure Example
|
||||
|
||||
Add one behavioral contract next to a route schema. APOPHIS can verify cross-route behavior, such as whether a resource created by one route is retrievable through another.
|
||||
@@ -19,6 +23,8 @@ Add one behavioral contract next to a route schema. APOPHIS can verify cross-rou
|
||||
**Route:**
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -29,7 +35,7 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
@@ -47,7 +53,7 @@ Expected
|
||||
response_code(GET /users/{response_body(this).id}) == 200
|
||||
|
||||
Observed
|
||||
GET /users/usr-123 returned 404
|
||||
GET /users/usr-7d865e returned 404
|
||||
|
||||
Why this matters
|
||||
The resource created by POST /users is not retrievable.
|
||||
@@ -80,6 +86,9 @@ apophis init --preset safe-ci
|
||||
|
||||
# 3. Verify
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
|
||||
# 4. Doctor
|
||||
apophis doctor
|
||||
```
|
||||
|
||||
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
||||
@@ -87,9 +96,12 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
||||
## Trust and Safety
|
||||
|
||||
- **Deterministic replay**: Every failure includes a seed and a one-command replay.
|
||||
- **Generation profile aliases**: Control test budget with `--generation-profile quick|standard|deep`.
|
||||
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
|
||||
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
|
||||
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
|
||||
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
|
||||
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
|
||||
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
|
||||
|
||||
## LLM-Safe
|
||||
@@ -116,4 +128,4 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
||||
|
||||
## License
|
||||
|
||||
ISC
|
||||
MIT
|
||||
|
||||
@@ -7,6 +7,8 @@ description: Use this skill when adding or improving APOPHIS contract-driven tes
|
||||
|
||||
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
|
||||
|
||||
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use this skill when the operator asks to:
|
||||
@@ -76,6 +78,7 @@ When entering a Fastify codebase:
|
||||
import Fastify from 'fastify'
|
||||
import swagger from '@fastify/swagger'
|
||||
import apophis from 'apophis-fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const app = Fastify()
|
||||
await app.register(swagger)
|
||||
@@ -114,8 +117,9 @@ app.post('/users', {
|
||||
}
|
||||
}
|
||||
}, async (req, reply) => {
|
||||
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||
reply.status(201)
|
||||
return { id: 'usr-1', ...req.body }
|
||||
return { id, ...req.body }
|
||||
})
|
||||
|
||||
await app.ready()
|
||||
@@ -350,6 +354,31 @@ Operator framing:
|
||||
|
||||
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
|
||||
|
||||
## Progressive Complexity
|
||||
|
||||
Start simple and add depth only where it pays off:
|
||||
|
||||
**Level 1 — Status and shape**: Every route gets an expected status code and key field existence.
|
||||
```apostl
|
||||
status:201
|
||||
response_body(this).id != null
|
||||
```
|
||||
|
||||
**Level 2 — Cross-route behavior**: Constructors check retrievability; mutators check persistence.
|
||||
```apostl
|
||||
response_code(GET /users/{response_body(this).id}) == 200
|
||||
response_body(GET /users/{response_body(this).id}).email == request_body(this).email
|
||||
```
|
||||
|
||||
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
|
||||
```apostl
|
||||
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
|
||||
```
|
||||
|
||||
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
|
||||
|
||||
Add level 2 before level 4. Do not skip level 2 for resource APIs.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
Do not:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# APOPHIS Plugin Contract System Specification
|
||||
|
||||
## Status: Active design; target version to be assigned
|
||||
## Status: Implemented
|
||||
|
||||
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
||||
## Operator Resources
|
||||
|
||||
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
|
||||
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout
|
||||
- [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
|
||||
|
||||
## CTAs
|
||||
|
||||
|
||||
@@ -144,7 +144,7 @@ See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
|
||||
`getToken` runs per request. Handle refresh inline:
|
||||
|
||||
```javascript
|
||||
let cachedToken: string | null = null
|
||||
let cachedToken = null
|
||||
|
||||
const auth = createAuthExtension({
|
||||
name: 'jwt-with-refresh',
|
||||
|
||||
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
|
||||
|
||||
| Environment | Cache | Reason |
|
||||
|-------------|-------|--------|
|
||||
| `production` | Disabled | No file I/O, no cache hits needed |
|
||||
| `test` | Disabled | Tests should be deterministic, no cache pollution |
|
||||
| `development` | Enabled | Speeds up iterative testing |
|
||||
| default | Enabled | Backward compatible |
|
||||
| `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `development` | Enabled by default | Speeds up iterative testing |
|
||||
| default | Enabled by default | Backward compatible |
|
||||
|
||||
## Cache Invalidation
|
||||
|
||||
|
||||
+21
-19
@@ -2,9 +2,11 @@
|
||||
|
||||
Inject controlled failures into contract tests to validate resilience guarantees.
|
||||
|
||||
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
||||
|
||||
## Usage
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const result = await fastify.apophis.contract({
|
||||
depth: 'standard',
|
||||
chaos: {
|
||||
@@ -14,7 +16,7 @@ const result = await fastify.apophis.contract({
|
||||
dropout: { probability: 1 },
|
||||
corruption: { probability: 1 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
## Event Types
|
||||
@@ -52,35 +54,35 @@ Mutates response bodies. Tests parsing robustness:
|
||||
response_body(this).id != null
|
||||
```
|
||||
|
||||
## Content-Type Aware Corruption
|
||||
## Corruption Strategies
|
||||
|
||||
Built-in strategies for common formats:
|
||||
Built-in strategies are content-type agnostic:
|
||||
|
||||
| Content-Type | Strategy | Effect |
|
||||
|-------------|----------|--------|
|
||||
| `application/json` | Truncate or null field | Removes fields or sets random field to null |
|
||||
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk |
|
||||
| `text/event-stream` | Event corrupt | Adds malformed SSE line |
|
||||
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
|
||||
| `text/plain` | Truncate | Cuts string in half |
|
||||
| Strategy | Effect |
|
||||
|----------|--------|
|
||||
| `truncate` | Cuts response body short |
|
||||
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
|
||||
| `field-corrupt` | Replaces a random field value with corrupted data |
|
||||
|
||||
Extension strategies can add content-type-specific behavior if needed.
|
||||
|
||||
## Custom Corruption via Extensions
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const myExtension = {
|
||||
name: 'custom-corrupt',
|
||||
corruptionStrategies: {
|
||||
'application/vnd.api+json': (data) => ({
|
||||
...data as object,
|
||||
...data,
|
||||
corrupted: true,
|
||||
}),
|
||||
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
||||
},
|
||||
}
|
||||
};
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [myExtension],
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
||||
@@ -90,7 +92,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
|
||||
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
||||
|
||||
```
|
||||
Error: Chaos mode is only available in test environment.
|
||||
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
|
||||
```
|
||||
|
||||
## Interpreting Results
|
||||
@@ -123,7 +125,7 @@ Failed tests include chaos events in diagnostics:
|
||||
|
||||
## Example: Testing Retry Logic
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
fastify.get('/data', {
|
||||
schema: {
|
||||
'x-ensures': [
|
||||
@@ -131,7 +133,7 @@ fastify.get('/data', {
|
||||
'redirect_count(this) <= 3',
|
||||
],
|
||||
},
|
||||
}, handler)
|
||||
}, handler);
|
||||
|
||||
// Test
|
||||
const result = await fastify.apophis.contract({
|
||||
@@ -139,5 +141,5 @@ const result = await fastify.apophis.contract({
|
||||
probability: 0.2,
|
||||
error: { probability: 1, statusCode: 503 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
+33
-4
@@ -10,15 +10,17 @@ Every command accepts these flags:
|
||||
|---|---|---|
|
||||
| `--config <path>` | Config file path | Auto-detect |
|
||||
| `--profile <name>` | Profile name from config | First profile |
|
||||
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived |
|
||||
| `--generation-profile <name>` | Generation budget profile (built-in: quick, standard, deep) | Depth-derived |
|
||||
| `--cwd <path>` | Working directory override | `process.cwd()` |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
|
||||
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
|
||||
| `--quiet` | Suppress non-error output | false |
|
||||
| `--verbose` | Enable verbose logging | false |
|
||||
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` |
|
||||
| `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
|
||||
| `--workspace` | Run supported commands across workspace packages | false |
|
||||
|
||||
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
|
||||
|
||||
## Commands
|
||||
|
||||
### `apophis init`
|
||||
@@ -37,8 +39,8 @@ apophis init --preset safe-ci
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `--force` | Overwrite existing files |
|
||||
| `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `-f, --force` | Overwrite existing files |
|
||||
| `--noninteractive` | Skip all prompts, require explicit flags |
|
||||
|
||||
**Examples:**
|
||||
@@ -64,6 +66,7 @@ apophis verify --profile quick --routes "POST /users"
|
||||
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||
| `--changed` | Filter to git-modified routes only |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
||||
|
||||
**Examples:**
|
||||
@@ -171,6 +174,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|
||||
|---|---|
|
||||
| `--mode <mode>` | Filter checks to a specific mode |
|
||||
| `--strict` | Treat warnings as failures |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
|
||||
**Checks:**
|
||||
|
||||
@@ -210,6 +214,31 @@ apophis migrate --dry-run
|
||||
apophis migrate --write
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### CI workflow with machine output
|
||||
```bash
|
||||
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
|
||||
```
|
||||
|
||||
### Monorepo workspace verification
|
||||
```bash
|
||||
apophis verify --workspace --profile quick
|
||||
apophis doctor --workspace
|
||||
```
|
||||
|
||||
### Replay a failure
|
||||
```bash
|
||||
apophis replay --artifact reports/apophis/failure-*.json
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `--changed` requires a git repository
|
||||
- `migrate` defaults to `--dry-run` (safe by default)
|
||||
- `--workspace` is only supported by `verify` and `doctor` commands
|
||||
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
@@ -40,7 +41,7 @@ fastify.post('/users', {
|
||||
}
|
||||
}
|
||||
}, async (req, reply) => {
|
||||
const id = `usr-${Date.now()}`
|
||||
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||
const user = { id, email: req.body.email, name: req.body.name }
|
||||
users.set(id, user)
|
||||
reply.status(201)
|
||||
|
||||
@@ -374,8 +374,10 @@ import { mkdirSync, rmSync } from 'fs'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
|
||||
let testCounter = 0
|
||||
|
||||
export function createTestWorkspace() {
|
||||
const dir = join(tmpdir(), `apophis-test-${Date.now()}`)
|
||||
const dir = join(tmpdir(), `apophis-test-${++testCounter}`)
|
||||
mkdirSync(dir, { recursive: true })
|
||||
|
||||
return {
|
||||
|
||||
+16
-106
@@ -2,6 +2,8 @@
|
||||
|
||||
Get from install to your first behavioral bug in 10 minutes.
|
||||
|
||||
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 20.x or 22.x
|
||||
@@ -30,6 +32,8 @@ This creates:
|
||||
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -40,27 +44,20 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
```
|
||||
|
||||
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead.
|
||||
|
||||
## Step 4: Run Verify
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
APOPHIS will:
|
||||
|
||||
1. Discover routes from your Fastify app
|
||||
2. Filter to `POST /users`
|
||||
3. Generate test data from the schema
|
||||
4. Execute the route
|
||||
5. Check the behavioral contract
|
||||
6. Print pass/fail, seed, and replay command
|
||||
|
||||
## Example Failure
|
||||
|
||||
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
||||
@@ -75,7 +72,7 @@ Expected
|
||||
response_code(GET /users/{response_body(this).id}) == 200
|
||||
|
||||
Observed
|
||||
GET /users/usr-123 returned 404
|
||||
GET /users/usr-7d865e returned 404
|
||||
|
||||
Why this matters
|
||||
The resource created by POST /users is not retrievable.
|
||||
@@ -100,111 +97,24 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
|
||||
## Next Steps
|
||||
|
||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
||||
- Run all routes: `apophis verify --profile quick`
|
||||
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
||||
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
|
||||
- Requires a git repository.
|
||||
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
||||
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
|
||||
|
||||
## Config Reference
|
||||
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
For the full configuration reference, see [CLI Reference](cli.md).
|
||||
|
||||
## Monorepo Workspaces
|
||||
|
||||
APOPHIS supports workspace-wide operations with the `--workspace` flag.
|
||||
|
||||
### Root package.json scripts
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"apophis:verify": "apophis verify --workspace --profile quick",
|
||||
"apophis:doctor": "apophis doctor --workspace",
|
||||
"apophis:qualify": "apophis qualify --workspace --profile ci"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Workspace fan-out
|
||||
|
||||
Run verify across all packages:
|
||||
Use `--workspace` to run verify or doctor across all packages:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output is package-attributed:
|
||||
|
||||
```json
|
||||
{
|
||||
"exitCode": 0,
|
||||
"runs": [
|
||||
{
|
||||
"package": "api",
|
||||
"cwd": "/repo/packages/api",
|
||||
"artifact": { ... }
|
||||
},
|
||||
{
|
||||
"package": "web",
|
||||
"cwd": "/repo/packages/web",
|
||||
"artifact": { ... }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Supported commands
|
||||
|
||||
- `apophis verify --workspace`
|
||||
- `apophis doctor --workspace`
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Success |
|
||||
| 1 | Behavioral / qualification failure |
|
||||
| 2 | Usage, config, or environment safety violation |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
See [CLI Reference](cli.md) for workspace output format and exit codes.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
|
||||
|
||||
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
|
||||
|
||||
## Why APOPHIS Is Good for LLM-Generated Services
|
||||
|
||||
Coding agents benefit from:
|
||||
@@ -18,10 +20,10 @@ Use `apophis init` with a preset:
|
||||
|
||||
| Preset | Use Case |
|
||||
|---|---|
|
||||
| `safe-ci` | General CI-safe setup |
|
||||
| `llm-safe` | Ultra-minimal for LLM-generated code |
|
||||
| `platform-observe` | Observe-mode policy and runtime drift reporting |
|
||||
| `protocol-lab` | Multi-step flows and stateful testing |
|
||||
| `safe-ci` | Minimal CI-safe preset (default) |
|
||||
| `llm-safe` | Minimal preset for LLM-generated codebases |
|
||||
| `platform-observe` | Production-ready with observe mode |
|
||||
| `protocol-lab` | Multi-step flow and stateful testing |
|
||||
|
||||
```bash
|
||||
apophis init --preset llm-safe
|
||||
@@ -108,6 +110,8 @@ export default {
|
||||
### Route Template with Behavioral Contract
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -134,7 +138,7 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
|
||||
+50
-3
@@ -2,6 +2,8 @@
|
||||
|
||||
Runtime visibility and drift detection without blocking by default.
|
||||
|
||||
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
|
||||
|
||||
## What Observe Does
|
||||
|
||||
`apophis observe` validates your runtime observe configuration:
|
||||
@@ -65,14 +67,38 @@ profiles: {
|
||||
}
|
||||
```
|
||||
|
||||
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema.
|
||||
The `platform-observe` preset enables sampling. Configure the rate explicitly:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'staging-observe': {
|
||||
mode: 'observe',
|
||||
preset: 'platform-observe',
|
||||
routes: [],
|
||||
sampling: 1.0 // 100% of requests observed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Staging vs Production
|
||||
|
||||
| Environment | Blocking | Sampling | Sink Required |
|
||||
|---|---|---|---|
|
||||
| Staging | No (default) | 10% | Yes |
|
||||
| Production | No (default) | 1% | Yes |
|
||||
| Staging | No (default) | 100% | Yes |
|
||||
| Production | No (default) | 100% | Yes |
|
||||
|
||||
Default is `1.0` (100%). Configure lower rates for production explicitly:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'prod-observe': {
|
||||
mode: 'observe',
|
||||
preset: 'platform-observe',
|
||||
routes: [],
|
||||
sampling: 0.1 // 10% of requests observed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## `--check-config` Flag
|
||||
|
||||
@@ -138,3 +164,24 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Sink Endpoint Configuration
|
||||
|
||||
Configure the reporting sink endpoint in your observe config:
|
||||
|
||||
```javascript
|
||||
observe: {
|
||||
sink: {
|
||||
endpoint: 'http://collector.internal:4318'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Monorepo Validation
|
||||
|
||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
|
||||
|
||||
## Mode Mismatch
|
||||
|
||||
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
|
||||
```
|
||||
|
||||
@@ -1,16 +1,18 @@
|
||||
# APOPHIS Protocol Extensions Specification
|
||||
|
||||
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature
|
||||
## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
|
||||
|
||||
## 1. Overview
|
||||
|
||||
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
|
||||
|
||||
APOPHIS is grounded in [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE) to the core invariant framework.
|
||||
|
||||
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
|
||||
|
||||
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
||||
|
||||
**Shipped in v2.x:**
|
||||
**Shipped in v2.0.0:**
|
||||
|
||||
- `contract({ variants })` for multi-header/media negotiation execution.
|
||||
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
|
||||
@@ -166,12 +168,15 @@ jwtExtension({
|
||||
The JWT extension maintains state across a test run:
|
||||
|
||||
```javascript
|
||||
interface JwtExtensionState {
|
||||
/** Track seen JTIs for replay detection */
|
||||
seenJtis: Set<string>
|
||||
/** Cached decoded JWTs */
|
||||
decodedCache: Map<string, DecodedJwt>
|
||||
}
|
||||
/**
|
||||
* JWT extension state across a test run.
|
||||
* @property {Set<string>} seenJtis - Track seen JTIs for replay detection
|
||||
* @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
|
||||
*/
|
||||
const jwtExtensionState = {
|
||||
seenJtis: new Set(),
|
||||
decodedCache: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 3.5 Example Contracts
|
||||
@@ -234,16 +239,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
|
||||
### 4.4 Implementation
|
||||
|
||||
```javascript
|
||||
interface TimeControl {
|
||||
/** Advance simulated time by milliseconds */
|
||||
advance(ms: number): void
|
||||
/** Set simulated time to specific timestamp */
|
||||
set(isoString: string): void
|
||||
/** Get current simulated time */
|
||||
now(): number
|
||||
/** Reset to real time */
|
||||
reset(): void
|
||||
}
|
||||
/**
|
||||
* Time control for deterministic testing.
|
||||
* @property {function(number): void} advance - Advance simulated time by milliseconds
|
||||
* @property {function(string): void} set - Set simulated time to specific ISO timestamp
|
||||
* @property {function(): number} now - Get current simulated time
|
||||
* @property {function(): void} reset - Reset to real time
|
||||
*/
|
||||
const timeControl = {
|
||||
advance(ms) { /* ... */ },
|
||||
set(isoString) { /* ... */ },
|
||||
now() { return Date.now(); },
|
||||
reset() { /* ... */ }
|
||||
};
|
||||
```
|
||||
|
||||
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
|
||||
@@ -288,11 +296,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
|
||||
Extension state tracks tokens across requests:
|
||||
|
||||
```javascript
|
||||
interface StatefulExtensionState {
|
||||
seenTokens: Set<string>
|
||||
consumedTokens: Set<string>
|
||||
categoryHistory: Map<string, EvalContext> // category -> last context
|
||||
}
|
||||
/**
|
||||
* Stateful extension state tracking tokens across requests.
|
||||
* @property {Set<string>} seenTokens - Tokens observed in previous requests
|
||||
* @property {Set<string>} consumedTokens - Tokens that have been consumed
|
||||
* @property {Map<string, EvalContext>} categoryHistory - category -> last context
|
||||
*/
|
||||
const statefulExtensionState = {
|
||||
seenTokens: new Set(),
|
||||
consumedTokens: new Set(),
|
||||
categoryHistory: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 5.4 Example Contracts
|
||||
@@ -522,14 +536,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
|
||||
## 14. Implementation Plan
|
||||
|
||||
### Phase 1: JWT + Time Control (P0)
|
||||
**Target**: v1.3.0
|
||||
### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/jwt.ts` — JWT extension implementation
|
||||
- `src/extensions/time.ts` — Time control extension
|
||||
- `src/extensions/stateful.ts` — Stateful predicates extension
|
||||
- `src/test/jwt-extension.test.ts` — JWT tests
|
||||
- `src/test/time-extension.test.ts` — Time control tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
- `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
|
||||
|
||||
**Tests**:
|
||||
- Decode Base64URL claims without verification
|
||||
@@ -539,27 +553,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
- `now()` predicate with mocked time
|
||||
- `apophis.time.advance()` in stateful tests
|
||||
|
||||
### Phase 2: X.509 + SPIFFE (P1)
|
||||
**Target**: v1.3.1
|
||||
### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/x509.ts` — X.509 extension
|
||||
- `src/extensions/spiffe.ts` — SPIFFE extension
|
||||
- `src/test/x509-extension.test.ts` — X.509 tests
|
||||
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 3: Token Hash + HTTP Signature (P2)
|
||||
**Target**: v1.3.2
|
||||
### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/token-hash.ts` — Token hash extension
|
||||
- `src/extensions/http-signature.ts` — HTTP signature extension
|
||||
- `src/test/token-hash-extension.test.ts` — Token hash tests
|
||||
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 4: Request Context (P2)
|
||||
**Target**: v1.3.3
|
||||
### Phase 4: Request Context (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/request-context.ts` — Request context predicates
|
||||
- `src/test/request-context-extension.test.ts` — Request context tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
---
|
||||
|
||||
|
||||
+74
-33
@@ -2,6 +2,8 @@
|
||||
|
||||
Run scenario, stateful, and chaos checks against non-production Fastify services.
|
||||
|
||||
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
|
||||
|
||||
## What Qualify Does
|
||||
|
||||
`apophis qualify` runs deeper testing than verify:
|
||||
@@ -49,6 +51,38 @@ profiles: {
|
||||
}
|
||||
```
|
||||
|
||||
## Scenario Definitions
|
||||
|
||||
Scenarios are multi-step flows with capture and rebind:
|
||||
|
||||
```javascript
|
||||
await fastify.apophis.scenario({
|
||||
name: 'oauth-basic',
|
||||
steps: [
|
||||
{
|
||||
name: 'authorize',
|
||||
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
|
||||
expect: ['status:200', 'response_payload(this).code != null'],
|
||||
capture: { code: 'response_payload(this).code' }
|
||||
},
|
||||
{
|
||||
name: 'token',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/oauth/token',
|
||||
form: { grant_type: 'authorization_code', code: '$authorize.code' }
|
||||
},
|
||||
expect: ['status:200', 'response_payload(this).access_token != null']
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
Scenario behavior:
|
||||
1. Cookie jar persists `Set-Cookie` values across steps.
|
||||
2. Step-level `headers.cookie` overrides jar values for that step.
|
||||
3. `form` sends `application/x-www-form-urlencoded` payloads.
|
||||
|
||||
## Stateful Testing
|
||||
|
||||
Stateful tests generate sequences of operations and track resources:
|
||||
@@ -58,7 +92,18 @@ Stateful tests generate sequences of operations and track resources:
|
||||
3. **Observer**: Read resources (GET)
|
||||
4. **Destructor**: Remove resources (DELETE)
|
||||
|
||||
APOPHIS automatically tracks created resources and cleans them up after testing.
|
||||
APOPHIS tracks created resources and runs cleanup after test completion.
|
||||
|
||||
Run stateful tests via the API:
|
||||
|
||||
```javascript
|
||||
const stateful = await fastify.apophis.stateful({ depth: 'standard', seed: 42 })
|
||||
console.log('Stateful tests:', stateful.summary)
|
||||
```
|
||||
|
||||
## Route Transparency
|
||||
|
||||
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
|
||||
|
||||
## Chaos and Adversity
|
||||
|
||||
@@ -67,7 +112,9 @@ Chaos testing injects controlled failures:
|
||||
- **Delay**: Slow responses
|
||||
- **Error**: Return error status codes
|
||||
- **Dropout**: Connection failures
|
||||
- **Corruption**: Malformed response bodies
|
||||
- **Truncate**: Truncated response bodies
|
||||
- **Malformed**: Invalid JSON or content-type
|
||||
- **Field-corrupt**: Random field mutation in response objects
|
||||
|
||||
Configure chaos in your preset:
|
||||
|
||||
@@ -84,36 +131,6 @@ presets: {
|
||||
}
|
||||
```
|
||||
|
||||
## Profile Examples
|
||||
|
||||
### oauth-nightly
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'oauth-nightly': {
|
||||
name: 'oauth-nightly',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### lifecycle-deep
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'lifecycle-deep': {
|
||||
name: 'lifecycle-deep',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Non-Prod Boundaries
|
||||
|
||||
Qualify mode is gated away from production by default:
|
||||
@@ -122,7 +139,7 @@ Qualify mode is gated away from production by default:
|
||||
|---|---|---|---|
|
||||
| local | enabled | enabled | enabled |
|
||||
| test/CI | enabled | enabled | enabled |
|
||||
| staging | enabled with allowlist | synthetic-only | canary-only |
|
||||
| staging | enabled with allowlist | enabled | blocked on protected routes |
|
||||
| production | disabled by default | disabled by default | disabled by default |
|
||||
|
||||
## Machine Output for CI
|
||||
@@ -224,3 +241,27 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Gate Execution Counts
|
||||
|
||||
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
|
||||
|
||||
## Zero-Execution Guardrail
|
||||
|
||||
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
|
||||
|
||||
## `--workspace` Flag
|
||||
|
||||
Run qualify across all packages in a monorepo workspace:
|
||||
|
||||
```bash
|
||||
apophis qualify --workspace --profile oauth-nightly
|
||||
```
|
||||
|
||||
## `--generation-profile` Flag
|
||||
|
||||
Control test data generation depth independently from the qualification profile:
|
||||
|
||||
```bash
|
||||
apophis qualify --profile oauth-nightly --generation-profile quick
|
||||
```
|
||||
|
||||
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
|
||||
**Symptoms**
|
||||
- `Unexpected token` in formula output
|
||||
- `Unterminated string` in x-ensures clause
|
||||
- `Unterminated string literal` in x-ensures clause
|
||||
- `Missing this` in operation call
|
||||
|
||||
**Resolution**
|
||||
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
**Symptoms**
|
||||
- `Plugin decorator already added`
|
||||
- `Duplicate route registration`
|
||||
- `No behavioral contracts found`
|
||||
- `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
|
||||
|
||||
**Resolution**
|
||||
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
|
||||
2. Check for multiple imports or plugin registrations in test vs production entry points.
|
||||
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas.
|
||||
3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
|
||||
4. Run `apophis doctor` to verify route discovery matches expectations.
|
||||
|
||||
**Prevention**
|
||||
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
|
||||
|
||||
```bash
|
||||
# Inspect the artifact
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
|
||||
|
||||
# Replay the exact failure
|
||||
apophis replay --artifact reports/apophis/verify-<timestamp>.json
|
||||
apophis replay --artifact reports/apophis/failure-<timestamp>.json
|
||||
|
||||
# Filter by error category
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+28
-34
@@ -2,15 +2,7 @@
|
||||
|
||||
Deterministic contract verification for CI and local development.
|
||||
|
||||
## What Verify Does
|
||||
|
||||
`apophis verify` runs behavioral contracts against your Fastify routes:
|
||||
|
||||
1. Discovers routes from your Fastify app
|
||||
2. Filters routes by profile config and CLI flags
|
||||
3. Generates test data from JSON Schema
|
||||
4. Executes routes and checks `x-ensures` contracts
|
||||
5. Reports pass/fail with deterministic seed and replay command
|
||||
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
|
||||
|
||||
## When to Use It
|
||||
|
||||
@@ -79,6 +71,8 @@ apophis verify --routes "POST /users/*"
|
||||
apophis verify --profile quick
|
||||
```
|
||||
|
||||
`*` and `?` wildcards are supported in `--routes`.
|
||||
|
||||
## `--changed` Flag
|
||||
|
||||
Run only routes modified in the current git branch:
|
||||
@@ -126,6 +120,8 @@ Next
|
||||
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||
```
|
||||
|
||||
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
|
||||
|
||||
## Machine Output for CI
|
||||
|
||||
Use concise formats to reduce log volume in large verify runs:
|
||||
@@ -137,6 +133,7 @@ Use concise formats to reduce log volume in large verify runs:
|
||||
|
||||
```bash
|
||||
# Extract only failed routes from full ndjson
|
||||
# Note: route.failed events are only emitted for failures, not passed routes
|
||||
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
|
||||
|
||||
# Write artifact to disk and parse the file instead of stdout
|
||||
@@ -149,7 +146,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
|---|---|
|
||||
| 0 | All contracts passed |
|
||||
| 1 | One or more behavioral contracts failed |
|
||||
| 2 | Config error or no routes matched |
|
||||
| 2 | Config error, no routes matched, no contracts found, or not a git repo |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
|
||||
@@ -158,42 +155,39 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
timeout: 5000
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
For the full config schema, see [CLI Reference](cli.md).
|
||||
|
||||
## Workspace Support
|
||||
|
||||
Run verify across all packages in a monorepo workspace:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output includes per-package pass/fail summaries. Fails if any package fails.
|
||||
|
||||
## `--generation-profile` Flag
|
||||
|
||||
Control test data generation depth independently from the verification profile:
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --generation-profile quick
|
||||
```
|
||||
|
||||
+2170
File diff suppressed because it is too large
Load Diff
@@ -7,13 +7,13 @@
|
||||
*
|
||||
* Inferred contracts are additive: they supplement, never replace, explicit x-ensures.
|
||||
*
|
||||
* Supported inference:
|
||||
* - required fields → response_body(this).field != null
|
||||
* - minimum (number/integer) → response_body(this).field >= N
|
||||
* - maximum (number/integer) → response_body(this).field <= N
|
||||
* - pattern (string) → response_body(this).field matches "..."
|
||||
* - const → response_body(this).field == value
|
||||
* - enum (small) → response_body(this).field == "a" || response_body(this).field == "b"
|
||||
* Supported inference (all wrapped with status-code guard):
|
||||
* - required fields → response_code(this) == N => response_body(this).field != null
|
||||
* - minimum (number/integer) → response_code(this) == N => response_body(this).field >= N
|
||||
* - maximum (number/integer) → response_code(this) == N => response_body(this).field <= N
|
||||
* - pattern (string) → response_code(this) == N => response_body(this).field matches "..."
|
||||
* - const → response_code(this) == N => response_body(this).field == value
|
||||
* - enum (small) → response_code(this) == N => response_body(this).field == "a" || ...
|
||||
*
|
||||
* Not inferred (leave to x-ensures for business logic):
|
||||
* - minLength/maxLength
|
||||
@@ -188,7 +188,12 @@ export function inferContractsFromRouteSchema(
|
||||
const code = parseInt(statusCode, 10)
|
||||
if (code >= 200 && code < 300) {
|
||||
const inferred = inferContractsFromResponseSchema(statusSchema)
|
||||
formulas.push(...inferred)
|
||||
// Wrap each inferred contract with a status-code guard so it only
|
||||
// applies when the response actually matches the schema it was
|
||||
// inferred from. Prevents a 200-schema const from failing on a 404.
|
||||
for (const formula of inferred) {
|
||||
formulas.push(`response_code(this) == ${code} => ${formula}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -753,3 +753,40 @@ test('integration: route-level x-variants are extracted and executed', async ()
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('integration: inferred contracts are guarded by status code', async () => {
|
||||
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||
try {
|
||||
await fastify.register(swagger, {})
|
||||
await fastify.register(apophisPlugin, {})
|
||||
fastify.get('/status-guarded', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
response: {
|
||||
200: {
|
||||
type: 'object',
|
||||
properties: { status: { type: 'string', const: 'success' } },
|
||||
required: ['status']
|
||||
},
|
||||
404: {
|
||||
type: 'object',
|
||||
properties: { error: { type: 'string' } },
|
||||
required: ['error']
|
||||
}
|
||||
}
|
||||
} as Record<string, unknown>
|
||||
}, async (request, reply) => {
|
||||
// Return 404 to verify the 200-schema const doesn't fail
|
||||
reply.status(404)
|
||||
return { error: 'not found' }
|
||||
})
|
||||
await fastify.ready()
|
||||
const result = await fastify.apophis.contract({ depth: 'quick' })
|
||||
// Should pass because the inferred const contract is guarded:
|
||||
// response_code(this) == 200 => response_body(this).status == "success"
|
||||
// The 404 response doesn't trigger the antecedent, so the implication holds.
|
||||
assert.strictEqual(result.summary.failed, 0, 'inferred 200-schema const should not fail on 404')
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user