Initial public release of Apophis — invariant-driven automated API testing

docs/qualify.md

@@ -2,6 +2,8 @@
 
 Run scenario, stateful, and chaos checks against non-production Fastify services.
 
+Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
+
 ## What Qualify Does
 
 `apophis qualify` runs deeper testing than verify:
@@ -49,6 +51,40 @@ profiles: {
 }
 ```
 
+## Scenario Definitions
+
+Scenarios are multi-step flows with capture and rebind:
+
+```javascript
+await fastify.apophis.scenario({
+  name: 'oauth-basic',
+    steps: [
+    {
+      name: 'authorize',
+      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
+      // Behavioral: state parameter round-trips for CSRF protection
+      expect: ['response_payload(this).state == request_query(this).state'],
+      capture: { code: 'response_payload(this).code' }
+    },
+    {
+      name: 'token',
+      request: {
+        method: 'POST',
+        url: '/oauth/token',
+        form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
+      },
+      // Behavioral: issued token preserves the requested scope
+      expect: ['response_payload(this).scope == request_body(this).scope']
+    }
+  ]
+})
+```
+
+Scenario behavior:
+1. Cookie jar persists `Set-Cookie` values across steps.
+2. Step-level `headers.cookie` overrides jar values for that step.
+3. `form` sends `application/x-www-form-urlencoded` payloads.
+
 ## Stateful Testing
 
 Stateful tests generate sequences of operations and track resources:
@@ -58,7 +94,18 @@ Stateful tests generate sequences of operations and track resources:
 3. **Observer**: Read resources (GET)
 4. **Destructor**: Remove resources (DELETE)
 
-APOPHIS automatically tracks created resources and cleans them up after testing.
+APOPHIS tracks created resources and runs cleanup after test completion.
+
+Run stateful tests via the API:
+
+```javascript
+const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
+console.log('Stateful tests:', stateful.summary)
+```
+
+## Route Transparency
+
+Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
 
 ## Chaos and Adversity
 
@@ -67,7 +114,9 @@ Chaos testing injects controlled failures:
 - **Delay**: Slow responses
 - **Error**: Return error status codes
 - **Dropout**: Connection failures
-- **Corruption**: Malformed response bodies
+- **Truncate**: Truncated response bodies
+- **Malformed**: Invalid JSON or content-type
+- **Field-corrupt**: Random field mutation in response objects
 
 Configure chaos in your preset:
 
@@ -84,36 +133,6 @@ presets: {
 }
 ```
 
-## Profile Examples
-
-### oauth-nightly
-
-```javascript
-profiles: {
-  'oauth-nightly': {
-    name: 'oauth-nightly',
-    mode: 'qualify',
-    preset: 'protocol-lab',
-    routes: [],
-    seed: 42
-  }
-}
-```
-
-### lifecycle-deep
-
-```javascript
-profiles: {
-  'lifecycle-deep': {
-    name: 'lifecycle-deep',
-    mode: 'qualify',
-    preset: 'protocol-lab',
-    routes: [],
-    seed: 42
-  }
-}
-```
-
 ## Non-Prod Boundaries
 
 Qualify mode is gated away from production by default:
@@ -122,7 +141,7 @@ Qualify mode is gated away from production by default:
 |---|---|---|---|
 | local | enabled | enabled | enabled |
 | test/CI | enabled | enabled | enabled |
-| staging | enabled with allowlist | synthetic-only | canary-only |
+| staging | enabled with allowlist | enabled | blocked on protected routes |
 | production | disabled by default | disabled by default | disabled by default |
 
 ## Machine Output for CI
@@ -186,7 +205,7 @@ export default {
   presets: {
     'protocol-lab': {
       name: 'protocol-lab',
-      depth: 'deep',
+    runs: 200,
       timeout: 15000,
       parallel: false,
       chaos: true,
@@ -224,3 +243,24 @@ export default {
   }
 };
 ```
+
+## Gate Execution Counts
+
+Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
+
+## Zero-Execution Guardrail
+
+Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
+
+## Test Budget
+
+The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
+
+```javascript
+presets: {
+  'protocol-lab': {
+    runs: 200,
+    timeout: 15000
+  }
+}
+```