docs: add paper citations, fix pedagogical issues, improve SKILL.md
- Cite arxiv 2602.23922 (Invariant-Driven Automated Testing) in all major docs - Add Progressive Complexity section to SKILL.md for LLM guidance - Fix SKILL.md Fast Start example to use deterministic ID generation - Fix getting-started.md failure output inconsistency - Fix auth-patterns.md TypeScript syntax in JS doc - Fix fastify-structure.md Date.now() in test helper - Fix observe.md misleading workspace heading - Build: clean | Tests: 849 pass, 0 fail
This commit is contained in:
@@ -4,6 +4,8 @@ Behavioral confidence for Fastify services.
|
|||||||
|
|
||||||
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
|
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
|
||||||
|
|
||||||
|
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
|
||||||
|
|
||||||
Supported Node.js versions: 20.x and 22.x.
|
Supported Node.js versions: 20.x and 22.x.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -7,6 +7,8 @@ description: Use this skill when adding or improving APOPHIS contract-driven tes
|
|||||||
|
|
||||||
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
|
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
|
||||||
|
|
||||||
|
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
|
||||||
|
|
||||||
## When To Use
|
## When To Use
|
||||||
|
|
||||||
Use this skill when the operator asks to:
|
Use this skill when the operator asks to:
|
||||||
@@ -76,6 +78,7 @@ When entering a Fastify codebase:
|
|||||||
import Fastify from 'fastify'
|
import Fastify from 'fastify'
|
||||||
import swagger from '@fastify/swagger'
|
import swagger from '@fastify/swagger'
|
||||||
import apophis from 'apophis-fastify'
|
import apophis from 'apophis-fastify'
|
||||||
|
import crypto from 'crypto'
|
||||||
|
|
||||||
const app = Fastify()
|
const app = Fastify()
|
||||||
await app.register(swagger)
|
await app.register(swagger)
|
||||||
@@ -114,8 +117,9 @@ app.post('/users', {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}, async (req, reply) => {
|
}, async (req, reply) => {
|
||||||
|
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||||
reply.status(201)
|
reply.status(201)
|
||||||
return { id: 'usr-1', ...req.body }
|
return { id, ...req.body }
|
||||||
})
|
})
|
||||||
|
|
||||||
await app.ready()
|
await app.ready()
|
||||||
@@ -350,6 +354,31 @@ Operator framing:
|
|||||||
|
|
||||||
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
|
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
|
||||||
|
|
||||||
|
## Progressive Complexity
|
||||||
|
|
||||||
|
Start simple and add depth only where it pays off:
|
||||||
|
|
||||||
|
**Level 1 — Status and shape**: Every route gets an expected status code and key field existence.
|
||||||
|
```apostl
|
||||||
|
status:201
|
||||||
|
response_body(this).id != null
|
||||||
|
```
|
||||||
|
|
||||||
|
**Level 2 — Cross-route behavior**: Constructors check retrievability; mutators check persistence.
|
||||||
|
```apostl
|
||||||
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
|
response_body(GET /users/{response_body(this).id}).email == request_body(this).email
|
||||||
|
```
|
||||||
|
|
||||||
|
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
|
||||||
|
```apostl
|
||||||
|
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
|
||||||
|
```
|
||||||
|
|
||||||
|
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
|
||||||
|
|
||||||
|
Add level 2 before level 4. Do not skip level 2 for resource APIs.
|
||||||
|
|
||||||
## Anti-Patterns
|
## Anti-Patterns
|
||||||
|
|
||||||
Do not:
|
Do not:
|
||||||
|
|||||||
@@ -144,7 +144,7 @@ See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
|
|||||||
`getToken` runs per request. Handle refresh inline:
|
`getToken` runs per request. Handle refresh inline:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
let cachedToken: string | null = null
|
let cachedToken = null
|
||||||
|
|
||||||
const auth = createAuthExtension({
|
const auth = createAuthExtension({
|
||||||
name: 'jwt-with-refresh',
|
name: 'jwt-with-refresh',
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
Inject controlled failures into contract tests to validate resilience guarantees.
|
Inject controlled failures into contract tests to validate resilience guarantees.
|
||||||
|
|
||||||
|
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
|||||||
@@ -374,8 +374,10 @@ import { mkdirSync, rmSync } from 'fs'
|
|||||||
import { tmpdir } from 'os'
|
import { tmpdir } from 'os'
|
||||||
import { join } from 'path'
|
import { join } from 'path'
|
||||||
|
|
||||||
|
let testCounter = 0
|
||||||
|
|
||||||
export function createTestWorkspace() {
|
export function createTestWorkspace() {
|
||||||
const dir = join(tmpdir(), `apophis-test-${Date.now()}`)
|
const dir = join(tmpdir(), `apophis-test-${++testCounter}`)
|
||||||
mkdirSync(dir, { recursive: true })
|
mkdirSync(dir, { recursive: true })
|
||||||
|
|
||||||
return {
|
return {
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
Get from install to your first behavioral bug in 10 minutes.
|
Get from install to your first behavioral bug in 10 minutes.
|
||||||
|
|
||||||
|
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- Node.js 20.x or 22.x
|
- Node.js 20.x or 22.x
|
||||||
@@ -70,7 +72,7 @@ Expected
|
|||||||
response_code(GET /users/{response_body(this).id}) == 200
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
|
|
||||||
Observed
|
Observed
|
||||||
GET /users/usr-123 returned 404
|
GET /users/usr-7d865e returned 404
|
||||||
|
|
||||||
Why this matters
|
Why this matters
|
||||||
The resource created by POST /users is not retrievable.
|
The resource created by POST /users is not retrievable.
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
|
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
|
||||||
|
|
||||||
|
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
|
||||||
|
|
||||||
## Why APOPHIS Is Good for LLM-Generated Services
|
## Why APOPHIS Is Good for LLM-Generated Services
|
||||||
|
|
||||||
Coding agents benefit from:
|
Coding agents benefit from:
|
||||||
|
|||||||
+4
-2
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
Runtime visibility and drift detection without blocking by default.
|
Runtime visibility and drift detection without blocking by default.
|
||||||
|
|
||||||
|
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
|
||||||
|
|
||||||
## What Observe Does
|
## What Observe Does
|
||||||
|
|
||||||
`apophis observe` validates your runtime observe configuration:
|
`apophis observe` validates your runtime observe configuration:
|
||||||
@@ -153,9 +155,9 @@ observe: {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Workspace Support
|
## Monorepo Validation
|
||||||
|
|
||||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages.
|
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
|
||||||
|
|
||||||
## Mode Mismatch
|
## Mode Mismatch
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,8 @@
|
|||||||
|
|
||||||
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
|
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
|
||||||
|
|
||||||
|
APOPHIS is grounded in [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE) to the core invariant framework.
|
||||||
|
|
||||||
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
|
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
|
||||||
|
|
||||||
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
Run scenario, stateful, and chaos checks against non-production Fastify services.
|
Run scenario, stateful, and chaos checks against non-production Fastify services.
|
||||||
|
|
||||||
|
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
|
||||||
|
|
||||||
## What Qualify Does
|
## What Qualify Does
|
||||||
|
|
||||||
`apophis qualify` runs deeper testing than verify:
|
`apophis qualify` runs deeper testing than verify:
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
Deterministic contract verification for CI and local development.
|
Deterministic contract verification for CI and local development.
|
||||||
|
|
||||||
|
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
|
||||||
|
|
||||||
## When to Use It
|
## When to Use It
|
||||||
|
|
||||||
- **Local development**: Quick feedback on behavioral changes
|
- **Local development**: Quick feedback on behavioral changes
|
||||||
|
|||||||
Reference in New Issue
Block a user