chore: crush git history - reborn from consolidation on 2026-03-10
This commit is contained in:
John Dvorak
@@ -0,0 +1,181 @@
|
||||
# Feedback for Apophis Team: Real-World Integration Challenges
|
||||
|
||||
## Context
|
||||
|
||||
We're integrating Apophis v1.1 into Arbiter, a multi-tenant identity platform with complex auth, graph-based permissions, and LinkedDataFragment responses. The goal is to use Apophis for contract testing of our Fastify routes.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
### 1. ~~APOSTL Syntax: Mandatory `else` Clause is Undocumented~~ ✅ FIXED in v2.0
|
||||
|
||||
**Status**: Resolved. APOPHIS v2.0 replaced APOSTL with Justin (plain JavaScript expressions).
|
||||
|
||||
**What we wrote (v1.x):**
|
||||
```apostl
|
||||
if response_code(this) == 201 then response_body(this).data.ok == true else T
|
||||
```
|
||||
|
||||
**What v2.0 uses:**
|
||||
```javascript
|
||||
statusCode == 201 ? response.body.data.ok == true : true
|
||||
// or simply:
|
||||
!(statusCode == 201) || response.body.data.ok == true
|
||||
```
|
||||
|
||||
**Resolution**: Justin uses standard JS ternary operators (`? :`) and boolean logic. No mandatory `else` clause, no custom syntax to learn.
|
||||
|
||||
### 2. Unclear Value Proposition vs Fastify Schema Validation
|
||||
|
||||
It took us time to understand what Apophis adds on top of Fastify's built-in JSON Schema validation.
|
||||
|
||||
**Fastify already provides:**
|
||||
- Request body/query/params validation (via Ajv)
|
||||
- Response serialization (via fast-json-stringify)
|
||||
- Error formatting
|
||||
|
||||
**We initially thought Apophis would:**
|
||||
- Validate responses against schemas (it doesn't — Fastify only serializes, doesn't validate responses)
|
||||
- Replace our need for separate test files (it partially does, but only for behavioral contracts)
|
||||
|
||||
**What Apophis actually adds:**
|
||||
- Behavioral contracts (`x-ensures`) for side effects and state changes
|
||||
- Property-based test generation from schemas
|
||||
- Stateful testing (constructor → mutator → destructor sequences)
|
||||
|
||||
**Suggestion:** Clarify in the "Getting Started" docs that Apophis is for *behavioral* contracts, not structural validation. Show a clear comparison table.
|
||||
|
||||
### 3. Testing Authenticated Routes is Underspecified
|
||||
|
||||
Our routes require:
|
||||
- JWT tokens in Authorization header
|
||||
- Tenant context (x-tenant-id header)
|
||||
- Permission checks via graph-based auth
|
||||
- Session cookies
|
||||
|
||||
**The problem:** Apophis generates requests programmatically, but there's no clear pattern for:
|
||||
- Injecting auth tokens into generated requests
|
||||
- Setting up prerequisite state (create user → login → get token → test route)
|
||||
- Handling token refresh or session management
|
||||
|
||||
**We tried:**
|
||||
- Using `scopes` to inject headers, but this is static and can't handle dynamic tokens
|
||||
- Using `x-requires` for preconditions, but it's unclear how to satisfy them
|
||||
|
||||
**Suggestion:** Document a pattern for authenticated routes. Examples:
|
||||
```typescript
|
||||
// Option 1: Dynamic scope setup
|
||||
await app.apophis.scope.register('authed', {
|
||||
headers: async () => ({
|
||||
'Authorization': `Bearer ${await getTestToken()}`
|
||||
})
|
||||
})
|
||||
|
||||
// Option 2: Test hooks
|
||||
await app.apophis.contract({
|
||||
beforeEach: async (req) => {
|
||||
req.headers['Authorization'] = await getTestToken()
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Running Against Real Server is Difficult
|
||||
|
||||
The docs show examples with inline route definitions, but we want to test our actual production routes.
|
||||
|
||||
**Challenges:**
|
||||
- Server bootstraps databases, WAL stores, ledger connections
|
||||
- Routes have complex dependency injection
|
||||
- We need to clean up between tests (file system conflicts, port binding)
|
||||
|
||||
**Example error:**
|
||||
```
|
||||
Error: EEXIST: file already exists, mkdir 'server-data/wal.log'
|
||||
```
|
||||
|
||||
**Suggestion:** Provide a guide for "Testing Existing Fastify Apps" that covers:
|
||||
- Bootstrapping the server in test mode
|
||||
- Cleaning up resources between runs
|
||||
- Configuring Apophis after server creation but before `ready()`
|
||||
|
||||
### 5. Contract Debugging is Hard
|
||||
|
||||
When contracts fail, the output is verbose but not actionable.
|
||||
|
||||
**Example output:**
|
||||
```json
|
||||
{
|
||||
"formula": "statusCode == 400 ? response.body.error != null : true",
|
||||
"context": {
|
||||
"expected": "non-null value",
|
||||
"actual": "undefined (field missing)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- We don't see the actual request that was generated
|
||||
- We don't see the full response body
|
||||
- No suggestion for how to fix the contract
|
||||
|
||||
**Suggestion:** Include in failure output:
|
||||
- The generated request (method, path, body)
|
||||
- The full response body
|
||||
- A suggestion like "Field 'error' missing from response. Check your handler returns error details."
|
||||
|
||||
### 6. No Clear CI/CD Pattern
|
||||
|
||||
We want to run Apophis in CI, but:
|
||||
- How do we handle database migrations/seeding?
|
||||
- How do we ensure deterministic runs (seed management)?
|
||||
- How do we fail the build on contract violations?
|
||||
|
||||
**Suggestion:** Add a CI/CD section to docs with GitHub Actions example that shows:
|
||||
```yaml
|
||||
- name: Contract Tests
|
||||
run: |
|
||||
npm run db:migrate:test
|
||||
npm run test:contracts
|
||||
# Exit code should be non-zero if contracts fail
|
||||
```
|
||||
|
||||
## What Works Well
|
||||
|
||||
- Schema-driven test generation is powerful
|
||||
- `x-category` auto-categorization reduces boilerplate
|
||||
- `check()` for single-route validation is useful
|
||||
- Integration with `@fastify/swagger` is seamless
|
||||
|
||||
## Recommendations
|
||||
|
||||
1. ~~**Make `else` optional** in APOSTL conditionals~~ ✅ Fixed in v2.0 — Justin uses standard JS ternary operators
|
||||
2. **Add "Auth Patterns" guide** with examples for JWT, sessions, API keys
|
||||
3. **Improve error messages** with request/response context and fix suggestions
|
||||
4. **Document real-world integration** (existing Fastify apps, not just toy examples)
|
||||
5. **Add CI/CD examples** with database setup and deterministic testing
|
||||
|
||||
## Our Current Workaround
|
||||
|
||||
We're using Apophis for:
|
||||
- Schema discovery and validation
|
||||
- Contract syntax checking
|
||||
- Documentation generation
|
||||
|
||||
But for authenticated routes, we're writing traditional E2E tests with `fastify.inject()` because we can control auth headers and setup/teardown more easily.
|
||||
|
||||
## Update: APOPHIS v2.0 Resolutions
|
||||
|
||||
**APOPHIS v2.0 (released 2026-04-25) addresses all feedback items:**
|
||||
|
||||
1. ✅ **APOSTL `else` clause**: Replaced with Justin (standard JS ternary `? :`)
|
||||
2. ✅ **Value proposition**: Documentation now clearly distinguishes structural vs behavioral validation
|
||||
3. ✅ **Auth patterns**: Extension system allows dynamic header injection via `onBuildRequest` hook
|
||||
4. ✅ **Real-world integration**: Guide added for testing existing Fastify apps with complex bootstrapping
|
||||
5. ✅ **Contract debugging**: Failure output now includes generated request, full response, and fix suggestions
|
||||
6. ✅ **CI/CD patterns**: GitHub Actions example with database migrations and deterministic seeds
|
||||
|
||||
**Recommended next steps for Arbiter integration:**
|
||||
- Migrate contracts from APOSTL to Justin using the [migration guide](docs/getting-started.md#migration-from-v1x)
|
||||
- Use the Extension Plugin System for Arbiter-specific predicates (`graph_check`, `partial_graph`, `budget_check`)
|
||||
- Register Arbiter extension to inject S2S headers and handle preflight/finalize lifecycle
|
||||
|
||||
See `docs/extensions/EXTENSION-PLUGIN-SYSTEM.md` for the Arbiter extension example.
|
||||
Reference in New Issue
Block a user