apophis-fastify/docs/qualify.md

Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

What Qualify Does

apophis qualify runs deeper testing than verify:

• Scenario execution: Multi-step protocol flows with capture/rebind
• Stateful testing: Constructor/mutator/observer/destructor sequences
• Chaos engineering: Controlled fault injection
• Adversity checks: Failure-path and edge-case validation

When to Use It

• Nightly CI: Scenario and stateful checks for critical flows
• Staging: Protocol flow validation before production
• Specialist teams: Auth, billing, workflow systems

Scenario Examples

OAuth Flow

profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Run with: apophis qualify --profile oauth-nightly --seed 42

Lifecycle Deep

profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. Constructor: Create resources (POST)
2. Mutator: Modify resources (PUT, PATCH)
3. Observer: Read resources (GET)
4. Destructor: Remove resources (DELETE)

APOPHIS automatically tracks created resources and cleans them up after testing.

Chaos and Adversity

Chaos testing injects controlled failures:

• Delay: Slow responses
• Error: Return error status codes
• Dropout: Connection failures
• Corruption: Malformed response bodies

Configure chaos in your preset:

presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}

Profile Examples

oauth-nightly

profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

lifecycle-deep

profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Non-Prod Boundaries

Qualify mode is gated away from production by default:

Environment	Scenario	Stateful	Chaos
local	enabled	enabled	enabled
test/CI	enabled	enabled	enabled
staging	enabled with allowlist	synthetic-only	canary-only
production	disabled by default	disabled by default	disabled by default

Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

Concise formats

• --format json-summary — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
• --format ndjson-summary — emits three NDJSON lines: run.started, run.summary, run.completed. No per-route events.

Filtering examples

# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis

Recommended CI retention strategy

• Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
• Use --artifact-dir to write artifacts automatically.
• Parse json-summary output for dashboards; keep full json artifacts for debugging.

Exit Codes

Code	Meaning
0	All qualifications passed
1	One or more qualifications failed
2	Safety violation or invalid config
3	Internal APOPHIS error
130	Interrupted (SIGINT)

Config Example

// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
      depth: 'deep',
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};