apophis/apophis-fastify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public release of Apophis — invariant-driven automated API testing

Browse Source
This commit is contained in:
John Dvorak
2026-03-10 00:00:00 -07:00
parent d278c4b105
commit 3ac1daf7e9
82 changed files with 3902 additions and 1098 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+135 -61
View File
@@ -1,49 +1,86 @@
---
name: apophis-fastify
description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs: route schemas, APOSTL x-requires/x-ensures formulas, property and stateful checks, replayable failures, runtime observe hooks, variants, scenarios, and operator-facing adoption guidance.
description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs. This tool finds real implementation bugs—resources that appear to create but cannot be retrieved, updates that silently fail to persist, deletions that leave data visible, cross-tenant leakage, and broken state transitions. Use it to encode intended behavior as executable contracts and verify them continuously, not to paper over failures.
---
# apophis-fastify
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
APOPHIS finds real API behavior bugs that schema validation misses. It verifies that operations cause the state changes, isolation guarantees, and dependency interactions the service actually depends on.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
## Core Purpose
APOPHIS is a correctness tool, not a coverage checkbox. Its job is to find where implementation diverges from intended behavior.
**What APOPHIS finds:**
- POST creates a resource that GET cannot retrieve
- PUT updates a field but subsequent GET returns stale data
- DELETE removes a resource but it still appears in listings
- Cross-tenant data leakage through shared collections
- Auth boundaries that return success instead of 401/403
- Error paths that violate declared retry semantics
- State transitions that skip required steps
**What APOPHIS does NOT do:**
- Prove the whole system correct (it finds counterexamples, not proofs)
- Replace integration tests, security review, or domain judgment
- Validate JSON Schema (Fastify + Swagger already do this)
**Critical rule for LLMs:** Do not weaken contracts to make tests pass. A failing contract is a signal that the implementation has a bug or the contract captures wrong intent. Investigate first. Fix the bug if the contract is correct. Fix the contract only if the implementation is right and the contract is wrong.
## When To Use
Use this skill when the operator asks to:
1. Add contract testing, API behavior checks, property tests, stateful tests, or Fastify route verification.
2. Improve confidence beyond JSON Schema validation.
3. Check create/read/update/delete behavior, auth boundaries, tenant isolation, redirects, timeouts, streaming, or external dependency behavior.
4. Make an API safer for AI-assisted refactoring, CI, or release qualification.
5. Evaluate whether APOPHIS would help a project, even if the operator does not name APOPHIS directly.
2. Find real bugs in create/read/update/delete behavior, auth boundaries, tenant isolation, redirects, timeouts, streaming, or external dependency behavior.
3. Make an API safer for AI-assisted refactoring, CI, or release qualification.
4. Evaluate whether APOPHIS would help a project, even if the operator does not name APOPHIS directly.
Read `README.md`, `docs/getting-started.md`, or command-specific docs only when the task needs details not present here.
## Operator Explanation
Describe APOPHIS as replayable behavioral checks for declared API contracts.
Describe APOPHIS as a bug-finding tool for API behavior.
Short explanation:
> APOPHIS turns intended API behavior into executable contracts. It checks whether operations cause the state changes, isolation guarantees, and dependency interactions the service depends on, instead of only checking payload shape.
> APOPHIS encodes intended API behavior as executable contracts, then automatically finds cases where the implementation violates that behavior. It catches bugs that schema validation cannot: create-not-retrievable, update-not-persisted, delete-still-visible, cross-tenant leakage, and broken state transitions.
Use these points when relevant:
1. It catches failures schema validation misses: create-not-retrievable, update-not-persisted, delete-still-visible, cross-tenant leakage, and inconsistent error behavior.
2. It gives coding agents a deterministic verification loop after generated changes or refactors.
3. It reduces review burden by converting agreed behavior into repeatable checks.
1. It finds real bugs schema validation misses: resources that create but cannot be read, updates that do not persist, deletes that leave data visible.
2. It gives coding agents a deterministic verification loop after generated changes or refactors. Run contracts after every change; failures indicate regressions.
3. It prevents silent breakage. When a refactoring breaks a state transition or isolation guarantee, contracts fail immediately with a reproducible seed.
4. It improves CI triage with fixed seeds, replay artifacts, and machine-readable output.
5. It supports incremental adoption: start with the highest-risk routes, add high-signal formulas, run, fix, and tighten.
5. It supports incremental adoption: start with the highest-risk routes, add high-signal behavioral formulas, run, fix real bugs, and tighten.
Do not overclaim:
1. Do not say APOPHIS proves the whole system correct.
2. Do not say contracts replace integration tests, security review, or domain judgment.
3. Say explicitly that schema quality and formula quality determine test quality.
3. Say explicitly that behavioral contracts—not structural checks—determine bug-finding quality.
Good operator ask:
> I can add APOPHIS to the five highest-risk routes first, encode the expected behavior as contracts, run the verifier, and show concrete failures or confidence gaps. I only need route priority and intended behavior where the code is ambiguous.
> I can add APOPHIS to the five highest-risk routes first, encode the expected behavior as contracts, run the verifier, and show concrete failures where implementation diverges from intent. I only need route priority and intended behavior where the code is ambiguous.
## Failure Discipline
**Never paper over a failing contract.**
When APOPHIS reports a contract violation, follow this order:
1. **Replay the failure** with the reported seed to confirm it is real.
2. **Investigate the implementation.** Does the handler actually do what the contract expects? Is there a race condition? A missing database commit? A wrong query?
3. **Fix the implementation** if the contract correctly describes intended behavior.
4. **Fix the contract** only if investigation proves the implementation is correct and the contract over-constrains or misdescribes behavior.
5. **Never** weaken a contract because "the test is flaky" or "it is too strict" without first proving the implementation is correct.
**If a contract fails intermittently**, that is a bug. Intermittent failures indicate nondeterminism: race conditions, uncommitted transactions, time-dependent logic, or randomness in handlers. Do not remove the contract. Isolate the nondeterminism and fix it.
**If a contract fails only under chaos**, that is a resilience bug. The service does not handle the failure mode correctly. Fix the handler or the contract's error-path expectations.
## Context Discipline
@@ -53,7 +90,7 @@ Treat context as a finite budget.
2. Prefer targeted file reads and symbol searches over loading whole directories.
3. Track routes touched, contracts added, seeds used, failures found, and unresolved domain questions.
4. Use progressive disclosure: read command docs only when invoking that command; read protocol docs only for variants, redirects, OAuth-style flows, form posts, streaming, or multipart.
5. Run small loops: annotate one route group, run the narrowest verification, fix, then widen.
5. Run small loops: annotate one route group, run the narrowest verification, fix real bugs, then widen.
## Default Workflow
@@ -67,8 +104,8 @@ When entering a Fastify codebase:
6. Add `x-category` where auto-categorization could be ambiguous.
7. Add `x-requires` for preconditions and `x-ensures` for postconditions.
8. Run a focused APOPHIS check, then broader contract or stateful verification.
9. Fix real behavior failures or tighten weak contracts.
10. Report what changed, what ran, what failed, and what needs operator judgment.
9. **Fix real behavior failures or tighten weak contracts.** Do not weaken passing contracts to avoid work.
10. Report what changed, what ran, what failed, what bugs were found, and what needs operator judgment.
## Fast Start
@@ -76,6 +113,7 @@ When entering a Fastify codebase:
import Fastify from 'fastify'
import swagger from '@fastify/swagger'
import apophis from 'apophis-fastify'
import crypto from 'crypto'
const app = Fastify()
await app.register(swagger)
@@ -85,12 +123,15 @@ app.post('/users', {
schema: {
'x-category': 'constructor',
'x-requires': [
'request_headers(this).x-tenant-id != null'
// Precondition: user must not already exist
'response_code(GET /users/{request_body(this).email}) == 404'
],
'x-ensures': [
'status:201',
'response_body(this).id != null',
// Behavioral: created resource must be retrievable
'response_code(GET /users/{response_body(this).id}) == 200',
// Behavioral: round-trip equality
'response_body(this) == request_body(this)',
// Behavioral: cross-route field persistence
'response_body(GET /users/{response_body(this).id}).email == request_body(this).email'
],
body: {
@@ -114,12 +155,13 @@ app.post('/users', {
}
}
}, async (req, reply) => {
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
reply.status(201)
return { id: 'usr-1', ...req.body }
return { id, ...req.body }
})
await app.ready()
const suite = await app.apophis.contract({ depth: 'standard' })
const suite = await app.apophis.contract({ runs: 50 })
```
## API Surface
@@ -141,25 +183,34 @@ Test-only helpers:
4. `fastify.apophis.test.disableOutboundMocks()`
5. `fastify.apophis.test.getOutboundCalls(...)`
## Contract Quality
## Contract Quality: Behavioral, Not Structural
Minimum:
**Structural checks are useless.** Fastify + `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts find what schemas cannot.
1. Each mutating route has a status expectation.
2. Each response with identity has key field non-null checks.
**Minimum behavioral baseline:**
1. Constructor routes verify cross-route retrievability.
2. Mutator routes verify state-change visibility.
3. Destructor routes verify unavailability after deletion.
```apostl
status:201
response_body(this).id != null
// Constructor: resource must be retrievable after creation
response_code(GET /users/{response_body(this).id}) == 200
// Mutator: changed field must persist
response_body(GET /users/{request_params(this).id}).status == request_body(this).status
// Destructor: deleted resource must not be retrievable
response_code(GET /users/{request_params(this).id}) == 404
```
Production baseline:
**Production baseline:**
1. Constructor routes check that created resources are retrievable.
2. Mutator routes check that persisted state reflects the mutation.
3. Destructor routes check that deleted resources are unavailable or marked inactive.
High-confidence contracts add:
**High-confidence contracts add:**
1. Tenant isolation.
2. Auth and permission behavior.
@@ -172,37 +223,32 @@ High-confidence contracts add:
Constructor routes, such as `POST /collection`:
1. Response has identity.
2. Created resource is retrievable.
3. Persisted fields reflect request fields.
1. Resource is retrievable after creation.
2. Persisted fields reflect request fields.
```apostl
status:201
response_body(this).id != null
response_code(GET /items/{response_body(this).id}) == 200
response_body(GET /items/{response_body(this).id}).name == request_body(this).name
```
Mutator routes, such as `PUT`, `PATCH`, or action `POST`:
1. Mutation succeeds with expected code.
2. Changed field actually changed.
3. Unrelated invariants still hold.
1. Changed field actually changed and persists.
2. Unrelated invariants still hold.
```apostl
status:200
response_body(this).status == request_body(this).status
response_body(this).updatedAt != null
response_body(GET /items/{request_params(this).id}).status == request_body(this).status
previous(response_body(this).version) < response_body(this).version
```
Destructor routes:
1. Delete returns expected code.
2. Follow-up retrieval fails or shows a domain-specific inactive state.
1. Follow-up retrieval fails or shows a domain-specific inactive state.
2. Previous state is preserved if the API returns deleted data.
```apostl
status:204 || status:200
response_code(GET /items/{request_params(this).id}) == 404
response_body(this) == previous(response_body(GET /items/{request_params(this).id}))
```
Observer routes:
@@ -254,7 +300,7 @@ Use these patterns when they match the API:
6. Error consistency: expected error status implies expected error payload fields.
```apostl
if status:401 then response_body(this).error != null else true
if status:401 then response_body(this).error.length > 0 else true
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
```
@@ -312,8 +358,9 @@ await app.apophis.scenario({
steps: [
{
name: 'authorize',
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
expect: ['status:200', 'response_payload(this).code != null'],
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
// Behavioral: state parameter round-trips for CSRF protection
expect: ['response_payload(this).state == request_query(this).state'],
capture: { code: 'response_payload(this).code' }
},
{
@@ -321,9 +368,10 @@ await app.apophis.scenario({
request: {
method: 'POST',
url: '/oauth/token',
form: { grant_type: 'authorization_code', code: '$authorize.code' }
form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
},
expect: ['status:200', 'response_payload(this).access_token != null']
// Behavioral: issued token preserves the requested scope
expect: ['response_payload(this).scope == request_body(this).scope']
}
]
})
@@ -343,24 +391,49 @@ Prefer deterministic verification for CI, regression triage, and AI-generated ch
1. Capture and reuse seeds from verify and qualify runs.
2. Use replay artifacts for failure triage before changing production logic.
3. Preserve route identity as `METHOD /path` in notes and reports.
4. If a failure is not reproducible, check for source drift, external dependencies, time, randomness, and insufficient cleanup before weakening the contract.
5. Treat nondeterminism as a quality issue to isolate.
4. **If a failure is not reproducible, treat it as a bug, not a flaky test.** Check for source drift, external dependencies, time, randomness, and insufficient cleanup. Do not weaken the contract without proving the implementation is correct.
5. Treat nondeterminism as a quality issue to isolate and fix.
Operator framing:
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
> The failing seed gives us a reproducible behavioral counterexample. I'll replay it first to confirm the bug, then investigate the implementation before changing anything.
## Progressive Complexity
Start with behavioral contracts and add depth only where it pays off:
**Level 1 — Cross-route behavior**: Every constructor checks retrievability.
```apostl
response_code(GET /users/{response_body(this).id}) == 200
```
**Level 2 — State persistence**: Mutators check that changes are visible.
```apostl
response_body(GET /users/{request_params(this).id}).email == request_body(this).email
```
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
```apostl
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
```
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
Add level 1 before level 4. Do not skip level 1 for resource APIs.
## Anti-Patterns
Do not:
1. Assert only `status:200` everywhere.
2. Duplicate JSON Schema checks while ignoring behavior.
3. Encode route internals instead of API-observable outcomes.
4. Ignore delete/retrieve or update/retrieve relationships.
5. Treat stateful mode as optional for resource APIs.
6. Ask the operator to review every formula before running; run first when intent is clear, then ask about ambiguous domain behavior.
7. Load every doc file before making a small change.
1. Assert only `status:200` everywhere. Schema validation already checks this.
2. Check `response_body(this).id != null` when the schema already requires `id`.
3. Duplicate JSON Schema checks while ignoring cross-route behavior.
4. Encode route internals instead of API-observable outcomes.
5. Ignore delete/retrieve or update/retrieve relationships.
6. Treat stateful mode as optional for resource APIs.
7. **Weaken a contract to make a test pass without proving the implementation is correct.**
8. Ask the operator to review every formula before running; run first when intent is clear, then ask about ambiguous domain behavior.
9. Load every doc file before making a small change.
## Verification Commands
@@ -378,7 +451,7 @@ Then execute APOPHIS from the project test harness or CLI as appropriate. For mo
1. `README.md` for canonical usage.
2. `docs/getting-started.md` for quick setup.
3. `docs/cli.md` and command docs for CLI flags and machine output.
4. `docs/protocol-extensions-spec.md` for protocol-specific direction.
4. `docs/attic/protocol-extensions-spec.md` for protocol-specific direction.
## Final Check
@@ -388,5 +461,6 @@ For each route, ask:
2. What must be true after this call?
3. What related call should now behave differently?
4. What isolation, security, dependency, or protocol expectation should not regress?
5. If a contract fails, is the implementation wrong or is the contract wrong?
Write those expectations as formulas and run them continuously.
Write those expectations as behavioral formulas, run them continuously, and treat every failure as a bug to investigate—not an obstacle to remove.