apophis-fastify/docs/qualify.md

# Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.

## What Qualify Does

`apophis qualify` runs deeper testing than verify:

- **Scenario execution**: Multi-step protocol flows with capture/rebind
- **Stateful testing**: Constructor/mutator/observer/destructor sequences
- **Chaos engineering**: Controlled fault injection
- **Adversity checks**: Failure-path and edge-case validation

## When to Use It

- **Nightly CI**: Scenario and stateful checks for critical flows
- **Staging**: Protocol flow validation before production
- **Specialist teams**: Auth, billing, workflow systems

## Scenario Examples

### OAuth Flow

```javascript
profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}
```

Run with: `apophis qualify --profile oauth-nightly --seed 42`

### Lifecycle Deep

```javascript
profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}
```

## Scenario Definitions

Scenarios are multi-step flows with capture and rebind:

```javascript
await fastify.apophis.scenario({
  name: 'oauth-basic',
  steps: [
    {
      name: 'authorize',
      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
      expect: ['status:200', 'response_payload(this).code != null'],
      capture: { code: 'response_payload(this).code' }
    },
    {
      name: 'token',
      request: {
        method: 'POST',
        url: '/oauth/token',
        form: { grant_type: 'authorization_code', code: '$authorize.code' }
      },
      expect: ['status:200', 'response_payload(this).access_token != null']
    }
  ]
})
```

Scenario behavior:
1. Cookie jar persists `Set-Cookie` values across steps.
2. Step-level `headers.cookie` overrides jar values for that step.
3. `form` sends `application/x-www-form-urlencoded` payloads.

## Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. **Constructor**: Create resources (POST)
2. **Mutator**: Modify resources (PUT, PATCH)
3. **Observer**: Read resources (GET)
4. **Destructor**: Remove resources (DELETE)

APOPHIS tracks created resources and runs cleanup after test completion.

Run stateful tests via the API:

```javascript
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)
```

## Route Transparency

Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.

## Chaos and Adversity

Chaos testing injects controlled failures:

- **Delay**: Slow responses
- **Error**: Return error status codes
- **Dropout**: Connection failures
- **Truncate**: Truncated response bodies
- **Malformed**: Invalid JSON or content-type
- **Field-corrupt**: Random field mutation in response objects

Configure chaos in your preset:

```javascript
presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}
```

## Non-Prod Boundaries

Qualify mode is gated away from production by default:

| Environment | Scenario | Stateful | Chaos |
|---|---|---|---|
| local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | enabled | blocked on protected routes |
| production | disabled by default | disabled by default | disabled by default |

## Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

### Concise formats

- `--format json-summary` — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
- `--format ndjson-summary` — emits three NDJSON lines: `run.started`, `run.summary`, `run.completed`. No per-route events.

### Filtering examples

```bash
# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis
```

### Recommended CI retention strategy

- Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
- Use `--artifact-dir` to write artifacts automatically.
- Parse `json-summary` output for dashboards; keep full `json` artifacts for debugging.

## Exit Codes

| Code | Meaning |
|---|---|
| 0 | All qualifications passed |
| 1 | One or more qualifications failed |
| 2 | Safety violation or invalid config |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |

## Config Example

```javascript
// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
    runs: 200,
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};
```

## Gate Execution Counts

Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.

## Zero-Execution Guardrail

Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.

## Test Budget

The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:

```javascript
presets: {
  'protocol-lab': {
    runs: 200,
    timeout: 15000
  }
}
```