apophis-fastify/docs/quality.md

Quality Engines

APOPHIS includes three quality engines for advanced testing: chaos injection, flake detection, and mutation testing. All require NODE_ENV=test.

Chaos Injection

Inject controlled failures into contract tests to validate resilience guarantees. Chaos events are generated by fast-check alongside test data, making them shrinkable — when a test fails, fast-check finds the minimal chaos event that causes the failure.

Usage

const result = await fastify.apophis.contract({
  runs: 50,
  chaos: {
    delay: { probability: 0.1, minMs: 100, maxMs: 500 },
    error: { probability: 0.1, statusCode: 503 },
    dropout: { probability: 0.05 },
    corruption: { probability: 0.1 },
  },
})

Event Types

Type	Effect	Tests
delay	Artificial latency	response_time(this) < 1000
error	Forces HTTP status code	Error-handling contracts
dropout	Network failure (status 0 or 504)	Fallback contracts
corruption	Mutates response bodies	Parsing robustness

Corruption Strategies

Strategy	Effect
truncate	Cuts response body in half
malformed	Returns invalid JSON ({"broken":)
field-corrupt	Sets a random field to null

Programmatic API

import {
  applyChaosToExecution,
  createChaosEventArbitrary,
  formatChaosEvents,
} from 'apophis-fastify'

// Apply pre-generated chaos events to a context
const result = applyChaosToExecution(ctx, events)

// Generate deterministic chaos events
const arb = createChaosEventArbitrary(config, contractNames)
const events = fc.sample(arb, { numRuns: 1, seed: 42 })[0]

// Format for diagnostics
console.log(formatChaosEvents(events))

Best Practices

1. Start small: probability: 0.05 (5% of requests)
2. Test one failure mode at a time
3. Verify contracts handle chaos: if status:503 then response_code(GET /health) == 200
4. Use seeds for reproducibility: seed: 42

Flake Detection

Automatically rerun failing tests with varied seeds to detect non-deterministic contracts. A "flake" is a test that fails on one run but passes on another with the same or different seed.

Usage

import { FlakeDetector } from 'apophis-fastify'

const detector = new FlakeDetector({
  sameSeedReruns: 1,    // Rerun with same seed
  seedVariations: 3,    // Try 3 additional seeds
})

const report = await detector.detectFlake(
  originalFailingResult,
  async (seed) => {
    const suite = await fastify.apophis.contract({ seed })
    return { passed: suite.summary.failed === 0 }
  },
  originalSeed
)

if (report.isFlaky) {
  console.log(`Flaky with ${report.confidence} confidence`)
  console.log('Reruns:', report.reruns)
}

Report Structure

{
  isFlaky: true,
  confidence: 'high', // 'high' | 'medium' | 'low'
  reruns: [
    { seed: 42, passed: false },
    { seed: 43, passed: true },
  ]
}

Confidence Scoring

Pass Rate	Confidence
0% pass	high (deterministic failure)
< 50% pass	medium
>= 50% pass	low (likely flaky)

Mutation Testing

Measure contract strength by injecting synthetic bugs. A "mutation" is a small change to a contract (e.g., flip == to !=). If the test suite catches the mutation (fails), the mutation is "killed". If it passes, the mutation "survives" — indicating weak coverage.

Usage

import { runMutationTesting } from 'apophis-fastify/quality/mutation'

const report = await runMutationTesting(fastify, {
  runs: 10,
  seed: 42,
  maxMutationsPerContract: 5,
  routes: ['/items'], // Optional: only test these routes
})

console.log(`Mutation score: ${report.score}%`)
console.log(`Killed: ${report.killed}, Survived: ${report.survived}`)
console.log('Weak contracts:', report.weakContracts)

Mutation Operators

Type	Example
flip-operator	== 201 → != 201
change-number	== 200 → == 201
remove-clause	A && B → A
negate-boolean	== true → == false
swap-variable	response_body → request_body
remove-ensures	Remove one ensures clause entirely

Report Structure

{
  score: 85,              // 0-100
  killed: 17,
  survived: 3,
  durationMs: 4500,
  weakContracts: ['POST /items'], // Routes where no mutations were killed
  mutations: [
    {
      mutation: {
        id: 'm0',
        route: 'POST /items',
        original: 'response_code(this) == 201',
        mutated: 'response_code(this) != 201',
        type: 'flip-operator',
      },
      killed: true,
      durationMs: 120,
    }
  ]
}

Single Mutation Test

Test a specific mutation without running the full suite:

import { testMutation } from 'apophis-fastify/quality/mutation'

const killed = await testMutation(fastify, contract, mutation, {
  runs: 10,
  seed: 42,
})

Environment Guard

All quality engines require NODE_ENV=test:

Error: chaos is only available in test environment.
Set NODE_ENV=test to enable quality features.

This prevents accidental execution in production or development.

Integration Example

Run all three engines in a CI pipeline:

// 1. Standard contract tests
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })

// 2. Chaos tests
const chaosSuite = await fastify.apophis.contract({
  runs: 50,
  seed: 42,
  chaos: { error: { probability: 0.1, statusCode: 503 } },
})

// 3. Flake detection on failures
for (const test of suite.tests.filter(t => !t.ok)) {
  const report = await detector.detectFlake(test, rerunFn, 42)
  if (report.isFlaky) {
    console.warn(`Flaky test detected: ${test.name}`)
  }
}

// 4. Mutation testing
const mutationReport = await runMutationTesting(fastify, { runs: 10 })
if (mutationReport.score < 80) {
  console.warn(`Low mutation score: ${mutationReport.score}%`)
}