apophis-fastify/docs/attic/testing-pyramid.md

APOPHIS Testing Pyramid

Overview

APOPHIS uses a three-layer testing pyramid. Unit tests form the base (most tests, fastest), integration tests sit in the middle, and end-to-end tests cap the pyramid (fewest tests, slowest). This document defines what belongs in each layer, how to decide where a new test goes, and the style rules all tests must follow.

---

Layer 1: Unit Tests (Bottom)

What belongs here

• Pure domain functions with no side effects: formula parser, formula evaluator, contract extraction, category inference, schema-to-arbitrary conversion, hash functions.
• Deterministic logic that accepts inputs and returns outputs.
• Property-based tests using fast-check that verify invariants of pure functions.

What does NOT belong here

• Fastify instance creation or HTTP injection.
• Database, file system, or network I/O.
• Tests that depend on process.env (unless the env is injected as a parameter).

How to decide

If the code under test can be imported and executed without Fastify(), without await fastify.ready(), and without touching the network, it belongs in a unit test.

Running time goal

< 10 ms per test.

Examples

• src/test/formula.test.ts — parser, evaluator, substitutor.
• src/test/domain.test.ts — category inference, contract extraction, route discovery with mock route arrays.
• src/test/incremental.test.ts — hashSchema, hashRoute.
• src/test/tap-formatter.test.ts — pure TAP string formatting.
• src/test/invariant-registry.test.ts — pure invariant checks against mock model state.
• src/test/resource-inference.test.ts — pure resource identity extraction.
• src/test/schema-to-arbitrary.test.ts — schema conversion and fast-check property tests.
• src/test/error-context.test.ts — contract validation with manually constructed EvalContext objects.
• src/test/cache-hints.test.ts — cache invalidation logic with mock routes.

---

Layer 2: Integration Tests (Middle)

What belongs here

• Plugin registration and decoration attachment on a Fastify instance.
• Route discovery using mocked route arrays (Fastify v5 does not expose routes directly).
• Scope registry auto-discovery from environment variables.
• Cleanup manager tracking and LIFO deletion.
• Hook validator registration (verify hooks attach without throwing).
• PETIT runner execution against a Fastify instance with mock routes and mocked dependencies.
• Stateful runner execution with mock routes.

What does NOT belong here

• Real external services (databases, message queues).
• Full HTTP lifecycle through all route handlers (that is E2E).
• Tests that take longer than 100 ms.

How to decide

If the test needs Fastify() and await fastify.ready() but does not need real HTTP requests to exercise the full handler chain, it is an integration test. Mock routes are preferred over real registered routes when the goal is to test discovery, categorization, or runner behavior.

Running time goal

< 100 ms per test.

Examples

• src/test/integration.test.ts — plugin registration, scope discovery, route discovery with mock routes, spec generation, PETIT runner, cleanup manager, hook validator.
• src/test/infrastructure.test.ts — scope registry, cleanup manager LIFO order, hook validator registration.
• src/test/stateful-runner.test.ts — stateful runner with mock routes.
• src/test/gap-fixes.test.ts — runtime validation hooks, previous() context, regex validation.
• src/test/scope-isolation.test.ts — scope filtering and header passing.

---

Layer 3: End-to-End Tests (Top)

What belongs here

• Full plugin + real routes + HTTP injection + contract validation.
• Tests that exercise the complete request lifecycle: preHandler hooks, handler execution, onResponse hooks, postcondition validation.
• Tests that verify the entire system works together: constructor → observer → mutator → cleanup.

What does NOT belong here

• Testing a single pure function (use unit tests).
• Testing plugin registration in isolation (use integration tests).
• Any test that can be written without fastify.inject().

How to decide

If the test needs real routes registered on Fastify, real handlers, and fastify.inject() to verify behavior across the full stack, it is an E2E test.

Running time goal

< 1 s per test.

Examples

• E2E tests are currently embedded in src/test/integration.test.ts and src/test/gap-fixes.test.ts. As the suite grows, consider splitting them into src/test/e2e/*.test.ts.

---

Test Placement Decision Tree

Does the test need Fastify?
  No  → Unit test
  Yes → Does it need real HTTP injection through handlers?
          No  → Integration test (mock routes OK)
          Yes → End-to-end test

---

Test Writing Best Practices

Arrange-Act-Assert (AAA)

Every test must have three distinct sections separated by blank lines:

1. Arrange — create inputs, set up mocks, construct context.
2. Act — call the function under test.
3. Assert — verify results using assert.

One assertion concept per test

A test should verify one behavior. Multiple assert calls are allowed if they check related properties of the same concept (e.g., verifying several fields of a returned object). Do not combine unrelated behaviors in a single test.

Descriptive test names

Use the should X when Y format:

• Good: should return utility category when path is /reset
• Bad: test category inference

No nested logic in tests

Avoid branching in example-based tests unless the branch is the behavior under test. Use helpers, table tests, or fast-check property tests for repeated cases.

Setup helpers for common fixtures

Create helper functions at the top of the test file for repeated setup:

const makeContext = (overrides: Partial<EvalContext> = {}): EvalContext => ({
  request: { body: null, headers: {}, query: {}, params: {}, cookies: {} },
  response: { body: null, headers: {}, statusCode: 200, responseTime: 0 },
  ...overrides,
} as EvalContext)

Cleanup resources

Every test that creates a Fastify instance must close it. Use try/finally if assertions might throw before the close call:

test('example', async () => {
  const fastify = Fastify()
  try {
    // arrange, act, assert
  } finally {
    await fastify.close()
  }
})

For tests that mutate process.env, save the original value and restore it:

const originalEnv = process.env
process.env = { ...originalEnv, FOO: 'bar' }
try {
  // test
} finally {
  process.env = originalEnv
}

Prefer strict equality assertions

Always use assert.strictEqual, assert.deepStrictEqual, and assert.notStrictEqual. Never use assert.equal or assert.deepEqual.

Property-based tests

Use fast-check for properties that must hold for all inputs:

test('property: generated integers respect bounds', async () => {
  await fc.assert(
    fc.property(fc.integer({ min: -1000, max: 1000 }), fc.integer({ min: -1000, max: 1000 }), (min, max) => {
      if (min > max) return true
      const schema = { type: 'integer', minimum: min, maximum: max }
      const arb = convertSchema(schema, { context: 'request' })
      const samples = fc.sample(arb, 100)
      return samples.every((n) => typeof n === 'number' && Number.isInteger(n) && n >= min && n <= max)
    })
  )
})

No summary documents

Do not create .md files to summarize test findings or work performed. All documentation belongs inline in code comments or in this testing pyramid document.

---

Cleanup Checklist for Test Authors

Before opening a PR, verify every test file you touch:

• Every Fastify() instance is closed with await fastify.close().
• If assertions might throw, the close is inside finally.
• process.env mutations are restored after the test.
• No event listeners are leaked (Fastify hooks are cleaned up on close).
• Cache or global state is reset if the test modifies it (invalidateCache() for cache tests).

---

Running Tests

# Run all tests
npm run test:src

# Run a specific file
npx tsc && node --test dist/test/formula.test.js