apophis-fastify/docs/attic/root-history/DX_IMPROVEMENT_PLAN.md

# APOPHIS DX Improvement Plan
## Getting Started, Error Context, Cache/CI Docs, and Human-Readable Output

---

## 1. GETTING STARTED GUIDE

### Goal
A complete "Hello World" to "Production Ready" guide that a developer can follow in 15 minutes.

### Structure

#### 1.1 Installation (30 seconds)
```bash
npm install apophis-fastify
# peer deps: fastify, @fastify/swagger
```

#### 1.2 Minimal Setup (2 minutes)
```typescript
import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify'

const fastify = Fastify()

// APOPHIS needs @fastify/swagger for spec generation
await fastify.register(import('@fastify/swagger'), {})
await fastify.register(apophisPlugin, {
  validateRuntime: true, // optional: validates contracts on every request
})

fastify.get('/health', {
  schema: {
    response: {
      200: {
        type: 'object',
        properties: { status: { type: 'string' } }
      }
    }
  }
}, async () => ({ status: 'ok' }))

await fastify.ready()

// Run contract tests
const result = await fastify.apophis.test({ mode: 'all', depth: 'quick' })
console.log(result.summary)
```

#### 1.3 Your First Contract (5 minutes)
Explain the mental model:
- **Requires** (preconditions): What must be true BEFORE the request
- **Ensures** (postconditions): What must be true AFTER the response
- **Invariants**: What must ALWAYS be true across requests

```typescript
fastify.post('/users', {
  schema: {
    'x-category': 'constructor',  // creates a resource
    'x-requires': [],             // no preconditions
    'x-ensures': [
      'status:201',
      'response_body(this).id != null',
      'response_body(this).email == request_body(this).email',
    ],
    body: {
      type: 'object',
      properties: {
        email: { type: 'string', format: 'email' },
        name: { type: 'string', minLength: 1 }
      },
      required: ['email', 'name']
    },
    response: {
      201: {
        type: 'object',
        properties: {
          id: { type: 'string' },
          email: { type: 'string' },
          name: { type: 'string' }
        }
      }
    }
  }
}, async (req, reply) => {
  reply.status(201)
  return { id: 'user-123', email: req.body.email, name: req.body.name }
})
```

#### 1.4 Complete CRUD Example (7 minutes)
Show a full resource lifecycle:
- POST /users (constructor)
- GET /users/:id (observer — reads the resource)
- PUT /users/:id (mutator — updates the resource)
- DELETE /users/:id (destructor — deletes the resource)

Demonstrate:
- How constructors populate the state
- How observers verify state
- How mutators maintain invariants
- How cleanup works

#### 1.5 Running in CI (1 minute)
```yaml
# .github/workflows/contracts.yml
- run: npm test
  env:
    APOPHIS_CHANGED_ROUTES: "${{ steps.changes.outputs.routes }}"
```

### Files to Create
- `docs/getting-started.md` — Full guide
- `docs/examples/crud-api.ts` — Complete working example
- `docs/examples/minimal.ts` — Single route example

---

## 2. RICH ERROR CONTEXT SYSTEM

### Current State (Bad)
```
Contract violation: response_body(this).id != null
```

No context. No request body. No response body. No status code. No suggestion.

### Target State (Good)
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 CONTRACT VIOLATION: POST /users
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Formula:
  response_body(this).id != null

Expected:
  id to be non-null

Actual:
  id = undefined

Request:
  POST /users
  Content-Type: application/json
  
  {
    "email": "alice@example.com",
    "name": "Alice"
  }

Response:
  HTTP/1.1 201 Created
  content-type: application/json
  
  {
    "email": "alice@example.com",
    "name": "Alice"
    // id is MISSING
  }

Suggestion:
  Your handler returned a 201 but forgot to include 'id' in the
  response body. Ensure your constructor routes return the created
  resource with its generated identifier.

Stack:
  at validatePostconditions (src/domain/contract-validation.ts:39)
  at runSequence (src/test/stateful-runner.ts:167)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

### Implementation Plan

#### Phase 1: Structured Error Objects
Replace string errors with rich error types:

```typescript
// src/types.ts
export interface ContractViolation {
  readonly type: 'contract-violation'
  readonly route: { method: string; path: string }
  readonly formula: string
  readonly formulaType: 'status' | 'apostl'
  readonly request: {
    body: unknown
    headers: Record<string, string>
    query: Record<string, unknown>
    params: Record<string, unknown>
  }
  readonly response: {
    statusCode: number
    headers: Record<string, string>
    body: unknown
  }
  readonly context: {
    expected: string
    actual: string
    diff?: string
  }
  readonly suggestion?: string
  readonly stack?: string
}
```

#### Phase 2: Smart Suggestions Engine
Add a suggestions module that maps common failures to actionable fixes:

```typescript
// src/domain/error-suggestions.ts
export const getSuggestion = (violation: ContractViolation): string | undefined => {
  // Status code mismatch
  if (violation.formulaType === 'status') {
    return `Expected status ${violation.context.expected}, got ${violation.context.actual}. Check your route handler's reply.status() call.`
  }
  
  // Null field
  if (violation.formula.includes('!= null') && violation.context.actual === 'undefined') {
    const field = extractField(violation.formula)
    return `Field '${field}' is missing from the response. Ensure your handler returns all required fields.`
  }
  
  // Equality mismatch
  if (violation.formula.includes('==')) {
    return `Expected values to match. Check for typos, case sensitivity, or missing transformations.`
  }
  
  // Authorization
  if (violation.formula.includes('authorization') || violation.formula.includes('tenant')) {
    return `This route may require authentication headers. Check your scope configuration.`
  }
  
  return undefined
}
```

#### Phase 3: Diff Generation
For equality comparisons, show a visual diff:

```typescript
// src/domain/error-formatter.ts
export const formatDiff = (expected: unknown, actual: unknown): string => {
  if (typeof expected === 'string' && typeof actual === 'string') {
    // String diff
    return `Expected: "${expected}"\nActual:   "${actual}"\nDiff:     ${generateCharDiff(expected, actual)}`
  }
  
  if (typeof expected === 'number' && typeof actual === 'number') {
    return `Expected: ${expected}\nActual:   ${actual}\nDelta:    ${actual - expected}`
  }
  
  // Object diff (shallow)
  return `Expected: ${JSON.stringify(expected, null, 2)}\nActual:   ${JSON.stringify(actual, null, 2)}`
}
```

#### Phase 4: Stack Traces
Capture the call stack at the point of failure:

```typescript
// In validatePostconditions
const stack = new Error().stack
return {
  success: false,
  error: new ContractViolation({
    // ... fields
    stack: cleanStack(stack),
  })
}
```

### Files to Create/Modify
- `src/types.ts` — Add `ContractViolation` interface
- `src/domain/error-suggestions.ts` — Suggestion engine
- `src/domain/error-formatter.ts` — Human-readable formatter
- `src/domain/contract-validation.ts` — Return structured errors
- `src/test/tap-formatter.ts` — Format violations in TAP output

---

## 3. CACHE/CI DOCUMENTATION

### Goal
Clear documentation for CI/CD integration with practical examples.

### Content

#### 3.1 Cache Overview
Explain:
- What gets cached (schema → arbitrary mappings, generated commands)
- Where it lives (`.apophis-cache.json` in project root)
- When it invalidates (schema hash mismatch, explicit hints)
- Performance impact (12x speedup on warm cache)

#### 3.2 CI/CD Integration Patterns

**Pattern A: Git-based Route Detection**
```bash
# Detect changed routes from git diff
CHANGED=$(git diff --name-only HEAD~1 | grep 'routes/' | sed 's|routes/||' | paste -sd ',' -)
APOPHIS_CHANGED_ROUTES="$CHANGED" npm test
```

**Pattern B: Manual Hints File**
```json
// .apophis-hints.json
{
  "changed": ["/users", "POST /orders"],
  "reason": "PR #123: Updated user and order endpoints"
}
```

**Pattern C: Full Cache Reset**
```bash
# Nuclear option: rebuild everything
rm .apophis-cache.json
npm test
```

**Pattern D: Monorepo Support**
```bash
# Per-package cache
APOPHIS_CACHE_FILE="./packages/api/.apophis-cache.json" npm test
```

#### 3.3 GitHub Actions Example
```yaml
name: Contract Tests

on: [push, pull_request]

jobs:
  contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Detect changed routes
        id: changes
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            CHANGED=$(git diff --name-only ${{ github.base_ref }} | grep -E 'routes/|schema/' || true)
            echo "routes=$CHANGED" >> $GITHUB_OUTPUT
          fi
      
      - name: Run contract tests
        run: npm test
        env:
          APOPHIS_CHANGED_ROUTES: ${{ steps.changes.outputs.routes }}
      
      - name: Upload cache artifact
        uses: actions/upload-artifact@v4
        with:
          name: apophis-cache
          path: .apophis-cache.json
```

#### 3.4 Cache Configuration API
```typescript
// Programmatic control
import { invalidateRoutes, invalidateCache } from 'apophis-fastify/incremental/cache'

// Before test run
invalidateRoutes(['/users'])  // Invalidate specific routes
invalidateCache()             // Clear everything
```

### Files to Create
- `docs/cache-and-ci.md` — Complete guide
- `docs/examples/github-actions.yml` — Working workflow
- `docs/examples/gitlab-ci.yml` — GitLab example

---

## 4. HUMAN-READABLE FAST-CHECK OUTPUT

### Current State (Bad)
```
Property failed after 42 tests
Counterexample: [{"name":"","email":"a@b.c"}]
```

### Target State (Good)
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 PROPERTY TEST FAILURE: POST /users
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Fast-check found a counterexample after 42 generated test cases:

Generated Input:
  {
    "name": "",           ← empty string (violates minLength: 1)
    "email": "a@b.c"      ← valid email format
  }

Request:
  POST /users
  Content-Type: application/json
  
  { "name": "", "email": "a@b.c" }

Response:
  HTTP/1.1 400 Bad Request
  
  { "error": "Name is required" }

Contract Violation:
  Postcondition: status:201
  Expected: 201 Created
  Actual:   400 Bad Request

Analysis:
  Your schema requires name to have minLength: 1, but the
  generated test case produced an empty string. Your handler
  correctly rejected it with 400, but the contract expects 201.
  
  Fix: Either:
    1. Remove minLength constraint from schema if empty names are valid
    2. Update contract to expect 400 for invalid input
    3. Add x-category: 'utility' if this is a validation endpoint

Shrunk: 3 times (from 128-character string to empty string)
Seed: 12345 (re-run with APOPHIS_SEED=12345 to reproduce)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

### Implementation Plan

#### Phase 1: Counterexample Formatter
```typescript
// src/test/counterexample-formatter.ts
export interface FormattedCounterexample {
  readonly route: { method: string; path: string }
  readonly generatedInput: Record<string, unknown>
  readonly request: { body: unknown; headers: Record<string, string> }
  readonly response: { statusCode: number; body: unknown }
  readonly contractViolation: ContractViolation
  readonly shrinkCount: number
  readonly seed: number
}

export const formatCounterexample = (example: FormattedCounterexample): string => {
  // Build human-readable output
}
```

#### Phase 2: Route Context in Errors
When fast-check finds a failure, include the route context:

```typescript
// In stateful-runner.ts, catch fast-check errors
try {
  await fc.assert(prop, { numRuns, seed })
} catch (err) {
  if (err instanceof fc.Error) {
    const formatted = formatFastCheckError(err, results)
    console.error(formatted)
  }
}
```

#### Phase 3: Analysis Engine
Auto-analyze failures and suggest fixes:

```typescript
// src/test/failure-analyzer.ts
export const analyzeFailure = (
  cmd: ApiOperation,
  ctx: EvalContext,
  violation: ContractViolation
): string => {
  // 400 status with 201 expectation
  if (ctx.response.statusCode === 400 && violation.formula === 'status:201') {
    return `Your handler rejected valid input. Check schema constraints match contract expectations.`
  }
  
  // Missing field
  if (violation.formula.includes('!= null') && violation.context.actual === 'undefined') {
    const field = extractField(violation.formula)
    return `Response missing '${field}'. Check your handler returns all required fields.`
  }
  
  // Schema mismatch
  return `Schema and contract may be out of sync. Review both for consistency.`
}
```

### Files to Create
- `src/test/counterexample-formatter.ts` — Format fast-check failures
- `src/test/failure-analyzer.ts` — Auto-analyze and suggest fixes
- `src/test/error-renderer.ts` — Terminal-friendly rendering with box drawing

---

## 5. ERROR SYSTEM ARCHITECTURE

### Design Principles
1. **Structured over String**: All errors are objects, not strings
2. **Context-Rich**: Every error includes request, response, and contract context
3. **Actionable**: Every error includes a suggestion for how to fix it
4. **Traceable**: Every error includes a stack trace and route identifier
5. **Diff-Friendly**: Equality failures show visual diffs
6. **Reproducible**: Every error includes the seed needed to reproduce

### Error Flow
```
Test Execution
    ↓
Contract Validation (contract-validation.ts)
    ↓
Structured Error Object (ContractViolation)
    ↓
Suggestion Engine (error-suggestions.ts)
    ↓
Diff Generation (error-formatter.ts)
    ↓
TAP Output (tap-formatter.ts)
    ↓
Console/CI Reporter
```

### Error Types
```typescript
export type ApophisError =
  | ContractViolation
  | FormulaParseError
  | FormulaEvalError
  | PreconditionError
  | InvariantError
  | TestGenerationError
```

---

## 6. IMPLEMENTATION ORDER

### Week 1: Foundation ✅ COMPLETE
- [x] Create `ContractViolation` type in `src/types.ts`
- [x] Update `contract-validation.ts` to return structured errors
- [x] Create `error-suggestions.ts` with basic suggestion engine
- [x] Update `tap-formatter.ts` to render rich diagnostics
- [x] Add tests for new error system
- [x] Fix `extractContract` null schema crash (`contract.ts:21`)
- [x] Fix `hashSchema` circular reference stack overflow (`hash.ts:24`)
- [x] Fix cleanup manager signal listener leak (`cleanup-manager.ts:48`)
- [x] Block dangerous accessors (`__proto__`, `constructor`, `prototype`) in formula evaluator
- [x] Normalize empty arrays to singletons in `extractContract`
- [x] Fix build output path (`tsconfig.json` rootDir)
- [x] Document route registration order requirement in README
- [x] Add violation deduplication in test output (PETIT + stateful runners)
- [x] Fix HEAD route noise in test generation
- [x] Add clean stack traces filtered to user code

**Status**: Error type chain tightened. `EvalResult` uses `error: string` with optional `violation?: ContractViolation`. Runners check `post.violation`. All 246 tests passing. Hardened against null schemas, circular references, prototype pollution, signal leaks, and duplicate failures.

### Week 2: Getting Started ✅ COMPLETE
- [x] Write `docs/getting-started.md`
- [x] Create `docs/examples/minimal.ts`
- [x] Create `docs/examples/crud-api.ts`
- [ ] Add screenshots/GIFs of test output
- [x] Update README with quick-start section

### Week 3: Cache/CI Docs
- [ ] Write `docs/cache-and-ci.md`
- [ ] Create GitHub Actions example
- [ ] Create GitLab CI example
- [ ] Document `APOPHIS_CHANGED_ROUTES`
- [ ] Document `.apophis-hints.json`

### Week 4: Fast-Check Formatter ✅ COMPLETE
- [x] Create `counterexample-formatter.ts`
- [x] Create `failure-analyzer.ts`
- [x] Create `error-renderer.ts` with box drawing
- [x] Integrate with stateful runner
- [x] Add tests for formatting

### Week 5: Production Hardening ✅ COMPLETE
- [x] Regex DoS protection with `safe-regex`
- [x] Standard logging with `pino` (APOPHIS_LOG_LEVEL)
- [x] Environment-aware cache (disabled in production/test)
- [x] Lazy cache loading (no sync file I/O at module load)
- [x] Fastify prefix support in route discovery
- [x] Signal handler deduplication (global Map)
- [x] Add `dispose()` method to CleanupManager
- [x] Remove all `console.log` from production code
- [x] Stryker mutation testing (contract-validation: 70%, error-suggestions: 68.7%)
- [x] Fix flaky property test (schema-to-arbitrary)
- [x] 345 tests passing

### Week 6: Scope Isolation ✅ COMPLETE
- [x] Implement scope filtering in `petit-runner.ts`
- [x] Implement scope filtering in `stateful-runner.ts`
- [x] Add scope headers to test requests via `buildRequest`
- [x] Tests for multi-scope scenarios

**Status**: Scope isolation fully implemented. Routes with `x-scope` annotation are filtered by the `scope` test parameter. Scope headers from `ScopeRegistry` are passed to test requests. 249 tests passing.

---

## 7. SUCCESS METRICS

- [ ] New user can go from `npm install` to passing contract tests in < 15 minutes
- [ ] Error messages include request/response context 100% of the time
- [ ] 80% of contract violations include an actionable suggestion
- [ ] CI integration documented for GitHub Actions, GitLab CI, and CircleCI
- [ ] Fast-check failures formatted with route context and analysis
- [ ] All examples in documentation are tested and working
- [ ] README has a "Getting Started" section above the fold