apophis-fastify/docs/attic/root-history/DX_IMPROVEMENT_PLAN.md

APOPHIS DX Improvement Plan

Getting Started, Error Context, Cache/CI Docs, and Human-Readable Output

---

1. GETTING STARTED GUIDE

Goal

A complete "Hello World" to "Production Ready" guide that a developer can follow in 15 minutes.

Structure

1.1 Installation (30 seconds)

npm install apophis-fastify
# peer deps: fastify, @fastify/swagger

1.2 Minimal Setup (2 minutes)

import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify'

const fastify = Fastify()

// APOPHIS needs @fastify/swagger for spec generation
await fastify.register(import('@fastify/swagger'), {})
await fastify.register(apophisPlugin, {
  validateRuntime: true, // optional: validates contracts on every request
})

fastify.get('/health', {
  schema: {
    response: {
      200: {
        type: 'object',
        properties: { status: { type: 'string' } }
      }
    }
  }
}, async () => ({ status: 'ok' }))

await fastify.ready()

// Run contract tests
const result = await fastify.apophis.test({ mode: 'all', depth: 'quick' })
console.log(result.summary)

1.3 Your First Contract (5 minutes)

Explain the mental model:

• Requires (preconditions): What must be true BEFORE the request
• Ensures (postconditions): What must be true AFTER the response
• Invariants: What must ALWAYS be true across requests

fastify.post('/users', {
  schema: {
    'x-category': 'constructor',  // creates a resource
    'x-requires': [],             // no preconditions
    'x-ensures': [
      'status:201',
      'response_body(this).id != null',
      'response_body(this).email == request_body(this).email',
    ],
    body: {
      type: 'object',
      properties: {
        email: { type: 'string', format: 'email' },
        name: { type: 'string', minLength: 1 }
      },
      required: ['email', 'name']
    },
    response: {
      201: {
        type: 'object',
        properties: {
          id: { type: 'string' },
          email: { type: 'string' },
          name: { type: 'string' }
        }
      }
    }
  }
}, async (req, reply) => {
  reply.status(201)
  return { id: 'user-123', email: req.body.email, name: req.body.name }
})

1.4 Complete CRUD Example (7 minutes)

Show a full resource lifecycle:

• POST /users (constructor)
• GET /users/:id (observer — reads the resource)
• PUT /users/:id (mutator — updates the resource)
• DELETE /users/:id (destructor — deletes the resource)

Demonstrate:

• How constructors populate the state
• How observers verify state
• How mutators maintain invariants
• How cleanup works

1.5 Running in CI (1 minute)

# .github/workflows/contracts.yml
- run: npm test
  env:
    APOPHIS_CHANGED_ROUTES: "${{ steps.changes.outputs.routes }}"

Files to Create

• docs/getting-started.md — Full guide
• docs/examples/crud-api.ts — Complete working example
• docs/examples/minimal.ts — Single route example

---

2. RICH ERROR CONTEXT SYSTEM

Current State (Bad)

Contract violation: response_body(this).id != null

No context. No request body. No response body. No status code. No suggestion.

Target State (Good)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 CONTRACT VIOLATION: POST /users
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Formula:
  response_body(this).id != null

Expected:
  id to be non-null

Actual:
  id = undefined

Request:
  POST /users
  Content-Type: application/json
  
  {
    "email": "alice@example.com",
    "name": "Alice"
  }

Response:
  HTTP/1.1 201 Created
  content-type: application/json
  
  {
    "email": "alice@example.com",
    "name": "Alice"
    // id is MISSING
  }

Suggestion:
  Your handler returned a 201 but forgot to include 'id' in the
  response body. Ensure your constructor routes return the created
  resource with its generated identifier.

Stack:
  at validatePostconditions (src/domain/contract-validation.ts:39)
  at runSequence (src/test/stateful-runner.ts:167)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Implementation Plan

Phase 1: Structured Error Objects

Replace string errors with rich error types:

// src/types.ts
export interface ContractViolation {
  readonly type: 'contract-violation'
  readonly route: { method: string; path: string }
  readonly formula: string
  readonly formulaType: 'status' | 'apostl'
  readonly request: {
    body: unknown
    headers: Record<string, string>
    query: Record<string, unknown>
    params: Record<string, unknown>
  }
  readonly response: {
    statusCode: number
    headers: Record<string, string>
    body: unknown
  }
  readonly context: {
    expected: string
    actual: string
    diff?: string
  }
  readonly suggestion?: string
  readonly stack?: string
}

Phase 2: Smart Suggestions Engine

Add a suggestions module that maps common failures to actionable fixes:

// src/domain/error-suggestions.ts
export const getSuggestion = (violation: ContractViolation): string | undefined => {
  // Status code mismatch
  if (violation.formulaType === 'status') {
    return `Expected status ${violation.context.expected}, got ${violation.context.actual}. Check your route handler's reply.status() call.`
  }
  
  // Null field
  if (violation.formula.includes('!= null') && violation.context.actual === 'undefined') {
    const field = extractField(violation.formula)
    return `Field '${field}' is missing from the response. Ensure your handler returns all required fields.`
  }
  
  // Equality mismatch
  if (violation.formula.includes('==')) {
    return `Expected values to match. Check for typos, case sensitivity, or missing transformations.`
  }
  
  // Authorization
  if (violation.formula.includes('authorization') || violation.formula.includes('tenant')) {
    return `This route may require authentication headers. Check your scope configuration.`
  }
  
  return undefined
}

Phase 3: Diff Generation

For equality comparisons, show a visual diff:

// src/domain/error-formatter.ts
export const formatDiff = (expected: unknown, actual: unknown): string => {
  if (typeof expected === 'string' && typeof actual === 'string') {
    // String diff
    return `Expected: "${expected}"\nActual:   "${actual}"\nDiff:     ${generateCharDiff(expected, actual)}`
  }
  
  if (typeof expected === 'number' && typeof actual === 'number') {
    return `Expected: ${expected}\nActual:   ${actual}\nDelta:    ${actual - expected}`
  }
  
  // Object diff (shallow)
  return `Expected: ${JSON.stringify(expected, null, 2)}\nActual:   ${JSON.stringify(actual, null, 2)}`
}

Phase 4: Stack Traces

Capture the call stack at the point of failure:

// In validatePostconditions
const stack = new Error().stack
return {
  success: false,
  error: new ContractViolation({
    // ... fields
    stack: cleanStack(stack),
  })
}

Files to Create/Modify

• src/types.ts — Add ContractViolation interface
• src/domain/error-suggestions.ts — Suggestion engine
• src/domain/error-formatter.ts — Human-readable formatter
• src/domain/contract-validation.ts — Return structured errors
• src/test/tap-formatter.ts — Format violations in TAP output

---

3. CACHE/CI DOCUMENTATION

Goal

Clear documentation for CI/CD integration with practical examples.

Content

3.1 Cache Overview

Explain:

• What gets cached (schema → arbitrary mappings, generated commands)
• Where it lives (.apophis-cache.json in project root)
• When it invalidates (schema hash mismatch, explicit hints)
• Performance impact (12x speedup on warm cache)

3.2 CI/CD Integration Patterns

Pattern A: Git-based Route Detection

# Detect changed routes from git diff
CHANGED=$(git diff --name-only HEAD~1 | grep 'routes/' | sed 's|routes/||' | paste -sd ',' -)
APOPHIS_CHANGED_ROUTES="$CHANGED" npm test

Pattern B: Manual Hints File

// .apophis-hints.json
{
  "changed": ["/users", "POST /orders"],
  "reason": "PR #123: Updated user and order endpoints"
}

Pattern C: Full Cache Reset

# Nuclear option: rebuild everything
rm .apophis-cache.json
npm test

Pattern D: Monorepo Support

# Per-package cache
APOPHIS_CACHE_FILE="./packages/api/.apophis-cache.json" npm test

3.3 GitHub Actions Example

name: Contract Tests

on: [push, pull_request]

jobs:
  contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Detect changed routes
        id: changes
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            CHANGED=$(git diff --name-only ${{ github.base_ref }} | grep -E 'routes/|schema/' || true)
            echo "routes=$CHANGED" >> $GITHUB_OUTPUT
          fi
      
      - name: Run contract tests
        run: npm test
        env:
          APOPHIS_CHANGED_ROUTES: ${{ steps.changes.outputs.routes }}
      
      - name: Upload cache artifact
        uses: actions/upload-artifact@v4
        with:
          name: apophis-cache
          path: .apophis-cache.json

3.4 Cache Configuration API

// Programmatic control
import { invalidateRoutes, invalidateCache } from 'apophis-fastify/incremental/cache'

// Before test run
invalidateRoutes(['/users'])  // Invalidate specific routes
invalidateCache()             // Clear everything

Files to Create

• docs/cache-and-ci.md — Complete guide
• docs/examples/github-actions.yml — Working workflow
• docs/examples/gitlab-ci.yml — GitLab example

---

4. HUMAN-READABLE FAST-CHECK OUTPUT

Current State (Bad)

Property failed after 42 tests
Counterexample: [{"name":"","email":"a@b.c"}]

Target State (Good)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 PROPERTY TEST FAILURE: POST /users
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Fast-check found a counterexample after 42 generated test cases:

Generated Input:
  {
    "name": "",           ← empty string (violates minLength: 1)
    "email": "a@b.c"      ← valid email format
  }

Request:
  POST /users
  Content-Type: application/json
  
  { "name": "", "email": "a@b.c" }

Response:
  HTTP/1.1 400 Bad Request
  
  { "error": "Name is required" }

Contract Violation:
  Postcondition: status:201
  Expected: 201 Created
  Actual:   400 Bad Request

Analysis:
  Your schema requires name to have minLength: 1, but the
  generated test case produced an empty string. Your handler
  correctly rejected it with 400, but the contract expects 201.
  
  Fix: Either:
    1. Remove minLength constraint from schema if empty names are valid
    2. Update contract to expect 400 for invalid input
    3. Add x-category: 'utility' if this is a validation endpoint

Shrunk: 3 times (from 128-character string to empty string)
Seed: 12345 (re-run with APOPHIS_SEED=12345 to reproduce)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Implementation Plan

Phase 1: Counterexample Formatter

// src/test/counterexample-formatter.ts
export interface FormattedCounterexample {
  readonly route: { method: string; path: string }
  readonly generatedInput: Record<string, unknown>
  readonly request: { body: unknown; headers: Record<string, string> }
  readonly response: { statusCode: number; body: unknown }
  readonly contractViolation: ContractViolation
  readonly shrinkCount: number
  readonly seed: number
}

export const formatCounterexample = (example: FormattedCounterexample): string => {
  // Build human-readable output
}

Phase 2: Route Context in Errors

When fast-check finds a failure, include the route context:

// In stateful-runner.ts, catch fast-check errors
try {
  await fc.assert(prop, { numRuns, seed })
} catch (err) {
  if (err instanceof fc.Error) {
    const formatted = formatFastCheckError(err, results)
    console.error(formatted)
  }
}

Phase 3: Analysis Engine

Auto-analyze failures and suggest fixes:

// src/test/failure-analyzer.ts
export const analyzeFailure = (
  cmd: ApiOperation,
  ctx: EvalContext,
  violation: ContractViolation
): string => {
  // 400 status with 201 expectation
  if (ctx.response.statusCode === 400 && violation.formula === 'status:201') {
    return `Your handler rejected valid input. Check schema constraints match contract expectations.`
  }
  
  // Missing field
  if (violation.formula.includes('!= null') && violation.context.actual === 'undefined') {
    const field = extractField(violation.formula)
    return `Response missing '${field}'. Check your handler returns all required fields.`
  }
  
  // Schema mismatch
  return `Schema and contract may be out of sync. Review both for consistency.`
}

Files to Create

• src/test/counterexample-formatter.ts — Format fast-check failures
• src/test/failure-analyzer.ts — Auto-analyze and suggest fixes
• src/test/error-renderer.ts — Terminal-friendly rendering with box drawing

---

5. ERROR SYSTEM ARCHITECTURE

Design Principles

1. Structured over String: All errors are objects, not strings
2. Context-Rich: Every error includes request, response, and contract context
3. Actionable: Every error includes a suggestion for how to fix it
4. Traceable: Every error includes a stack trace and route identifier
5. Diff-Friendly: Equality failures show visual diffs
6. Reproducible: Every error includes the seed needed to reproduce

Error Flow

Test Execution
    ↓
Contract Validation (contract-validation.ts)
    ↓
Structured Error Object (ContractViolation)
    ↓
Suggestion Engine (error-suggestions.ts)
    ↓
Diff Generation (error-formatter.ts)
    ↓
TAP Output (tap-formatter.ts)
    ↓
Console/CI Reporter

Error Types

export type ApophisError =
  | ContractViolation
  | FormulaParseError
  | FormulaEvalError
  | PreconditionError
  | InvariantError
  | TestGenerationError

---

6. IMPLEMENTATION ORDER

Week 1: Foundation ✅ COMPLETE

• Create ContractViolation type in src/types.ts
• Update contract-validation.ts to return structured errors
• Create error-suggestions.ts with basic suggestion engine
• Update tap-formatter.ts to render rich diagnostics
• Add tests for new error system
• Fix extractContract null schema crash (contract.ts:21)
• Fix hashSchema circular reference stack overflow (hash.ts:24)
• Fix cleanup manager signal listener leak (cleanup-manager.ts:48)
• Block dangerous accessors (__proto__, constructor, prototype) in formula evaluator
• Normalize empty arrays to singletons in extractContract
• Fix build output path (tsconfig.json rootDir)
• Document route registration order requirement in README
• Add violation deduplication in test output (PETIT + stateful runners)
• Fix HEAD route noise in test generation
• Add clean stack traces filtered to user code

Status: Error type chain tightened. EvalResult uses error: string with optional violation?: ContractViolation. Runners check post.violation. All 246 tests passing. Hardened against null schemas, circular references, prototype pollution, signal leaks, and duplicate failures.

Week 2: Getting Started ✅ COMPLETE

• Write docs/getting-started.md
• Create docs/examples/minimal.ts
• Create docs/examples/crud-api.ts
• Add screenshots/GIFs of test output
• Update README with quick-start section

Week 3: Cache/CI Docs

• Write docs/cache-and-ci.md
• Create GitHub Actions example
• Create GitLab CI example
• Document APOPHIS_CHANGED_ROUTES
• Document .apophis-hints.json

Week 4: Fast-Check Formatter ✅ COMPLETE

• Create counterexample-formatter.ts
• Create failure-analyzer.ts
• Create error-renderer.ts with box drawing
• Integrate with stateful runner
• Add tests for formatting

Week 5: Production Hardening ✅ COMPLETE

• Regex DoS protection with safe-regex
• Standard logging with pino (APOPHIS_LOG_LEVEL)
• Environment-aware cache (disabled in production/test)
• Lazy cache loading (no sync file I/O at module load)
• Fastify prefix support in route discovery
• Signal handler deduplication (global Map)
• Add dispose() method to CleanupManager
• Remove all console.log from production code
• Stryker mutation testing (contract-validation: 70%, error-suggestions: 68.7%)
• Fix flaky property test (schema-to-arbitrary)
• 345 tests passing

Week 6: Scope Isolation ✅ COMPLETE

• Implement scope filtering in petit-runner.ts
• Implement scope filtering in stateful-runner.ts
• Add scope headers to test requests via buildRequest
• Tests for multi-scope scenarios

Status: Scope isolation fully implemented. Routes with x-scope annotation are filtered by the scope test parameter. Scope headers from ScopeRegistry are passed to test requests. 249 tests passing.

---

7. SUCCESS METRICS

• New user can go from npm install to passing contract tests in < 15 minutes
• Error messages include request/response context 100% of the time
• 80% of contract violations include an actionable suggestion
• CI integration documented for GitHub Actions, GitLab CI, and CircleCI
• Fast-check failures formatted with route context and analysis
• All examples in documentation are tested and working
• README has a "Getting Started" section above the fold