chore: crush git history - reborn from consolidation on 2026-03-10
This commit is contained in:
John Dvorak
@@ -0,0 +1,393 @@
|
||||
## Feedback: Protocol Conformance and Bilingual Representation Testing
|
||||
|
||||
Status: Feedback from Arbiter integration work
|
||||
Date: 2026-04-27
|
||||
|
||||
## Context
|
||||
|
||||
We have been extending APOPHIS across Arbiter route families successfully for resource-oriented APIs:
|
||||
|
||||
1. Billing routes
|
||||
2. User directory routes
|
||||
3. Device management routes
|
||||
|
||||
That work went well once we moved to explicit schemas, explicit `x-ensures`, and avoided schema helpers that hard-coded one response shape.
|
||||
|
||||
Where things got much harder was OAuth 2.1.
|
||||
|
||||
The issue is not that OAuth is "too complex to test". The issue is that OAuth is a protocol with:
|
||||
|
||||
1. multiple representations for the same endpoint
|
||||
2. cross-step state transfer
|
||||
3. redirects, cookies, and form-encoded requests
|
||||
4. wire-level requirements that must remain spec-compliant by default
|
||||
|
||||
In Arbiter, OAuth endpoints must stay bilingual:
|
||||
|
||||
1. plain JSON by default for RFC compliance
|
||||
2. LDF only when explicitly requested via `Accept`
|
||||
|
||||
Today, APOPHIS pushes us toward a single response shape per route contract. That works well for resource APIs, but it creates pressure to distort protocol endpoints just to make them fit the contract runner.
|
||||
|
||||
The key outcome we want is:
|
||||
|
||||
APOPHIS should let us test rich protocols without forcing us to change compliant production behavior.
|
||||
|
||||
## What Already Works Well
|
||||
|
||||
These existing capabilities are the right building blocks:
|
||||
|
||||
1. `request_headers(this)`, `response_headers(this)`, `cookies(this)`
|
||||
2. `redirect_count(this)`, `redirect_url(this).0`, `redirect_status(this).0`
|
||||
3. stateful testing
|
||||
4. protocol extensions roadmap in `docs/protocol-extensions-spec.md`
|
||||
5. outbound mocking and deterministic seeded execution
|
||||
|
||||
This feedback is not asking for a rewrite. It is asking for a thin layer that composes these pieces into a protocol-testing model.
|
||||
|
||||
## Core Gap
|
||||
|
||||
APOPHIS currently fits best when a route has one canonical success body shape and one canonical error body shape.
|
||||
|
||||
OAuth 2.1 does not look like that:
|
||||
|
||||
1. `POST /oauth/token` is plain JSON by default
|
||||
2. the same endpoint may also return LDF when `Accept: application/ldf+json`
|
||||
3. `GET /oauth/authorize` often returns redirects instead of bodies
|
||||
4. multi-step flows pass state via cookies, redirect query params, auth codes, refresh tokens, and headers
|
||||
|
||||
The problem is not only schema generation. The deeper problem is that APOPHIS lacks a first-class way to say:
|
||||
|
||||
1. run the same route under multiple negotiated representations
|
||||
2. assert on the semantic payload independent of representation
|
||||
3. capture values from one step and feed them into later steps
|
||||
4. test a protocol scenario without replacing the route's default wire behavior
|
||||
|
||||
## Recommended Changes
|
||||
|
||||
### 1. Add Representation-Aware Contracts
|
||||
|
||||
Routes need multiple contract variants for the same endpoint.
|
||||
|
||||
Example need:
|
||||
|
||||
1. default `Accept: application/json` -> plain OAuth JSON
|
||||
2. explicit `Accept: application/ldf+json` -> LDF fragment wrapping the same semantic payload
|
||||
|
||||
Suggested direction:
|
||||
|
||||
Add a route-level annotation for negotiated variants, for example:
|
||||
|
||||
```ts
|
||||
schema: {
|
||||
'x-variants': [
|
||||
{
|
||||
name: 'json',
|
||||
when: 'request_headers(this).accept == null || request_headers(this).accept matches /application\/json/',
|
||||
response: {
|
||||
200: { type: 'object', properties: { access_token: { type: 'string' } } },
|
||||
400: { type: 'object', properties: { error: { type: 'string' } } }
|
||||
},
|
||||
ensures: [
|
||||
'if status:200 then response_body(this).access_token != null else true'
|
||||
]
|
||||
},
|
||||
{
|
||||
name: 'ldf',
|
||||
when: 'request_headers(this).accept matches /application\/(ldf\+json|vnd\.ldf\+json)/',
|
||||
response: {
|
||||
200: { type: 'object', properties: { type: { const: "LinkedDataFragment" }, fragment_type: { const: "Document" }, data: { type: 'object' } } }
|
||||
},
|
||||
ensures: [
|
||||
'if status:200 then response_body(this).fragment_type == "Document" else true'
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This would let one route remain spec-compliant by default while still being richly testable under negotiated formats.
|
||||
|
||||
### 2. Add a Semantic Payload Accessor
|
||||
|
||||
This is the smallest feature with the biggest payoff.
|
||||
|
||||
Today, formulas need to know whether the body is:
|
||||
|
||||
1. raw JSON: `response_body(this).access_token`
|
||||
2. LDF: `response_body(this).data.access_token`
|
||||
|
||||
That is exactly the wrong abstraction boundary for bilingual endpoints.
|
||||
|
||||
Suggested addition:
|
||||
|
||||
1. `response_payload(this)`
|
||||
|
||||
Semantics:
|
||||
|
||||
1. if body is an LDF fragment with `data`, return `body.data`
|
||||
2. otherwise return `body`
|
||||
|
||||
Then the same formula works for both representations:
|
||||
|
||||
```apostl
|
||||
if status:200 then response_payload(this).access_token != null else true
|
||||
if status:400 then response_payload(this).error == "unsupported_grant_type" else true
|
||||
```
|
||||
|
||||
Keep `response_body(this)` exactly as it is. `response_payload(this)` is the normalized semantic view.
|
||||
|
||||
This single feature would dramatically reduce contract duplication for negotiated responses.
|
||||
|
||||
### 3. Add Variant Execution to `contract()`
|
||||
|
||||
The test runner should be able to run the same route under multiple header sets.
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```ts
|
||||
await fastify.apophis.contract({
|
||||
depth: 'quick',
|
||||
routes: ['POST /oauth/token'],
|
||||
variants: [
|
||||
{ name: 'json', headers: { accept: 'application/json' } },
|
||||
{ name: 'ldf', headers: { accept: 'application/ldf+json' } }
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
This should:
|
||||
|
||||
1. reuse existing scope/header logic
|
||||
2. report failures per route per variant
|
||||
3. not require separate route registrations or test harnesses
|
||||
|
||||
This is much more useful than forcing a route to always return one representation.
|
||||
|
||||
### 4. Add a Protocol Scenario Runner
|
||||
|
||||
The docs currently say protocol state machines are out of scope and should use separate integration tests.
|
||||
|
||||
We think this boundary is too strict.
|
||||
|
||||
Not everything about OAuth needs to be declarative, but APOPHIS should still own the execution model for protocol scenarios.
|
||||
|
||||
What is needed is not a third giant testing engine. It is a thin scripted layer over the existing HTTP executor, formula evaluator, flake detection, state handling, and extensions.
|
||||
|
||||
Suggested API:
|
||||
|
||||
```ts
|
||||
await fastify.apophis.scenario({
|
||||
name: 'oauth21.refresh_rotation',
|
||||
steps: [
|
||||
{
|
||||
name: 'login',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/end-user/login',
|
||||
body: { userKey: 'u1', password: 'pw' },
|
||||
headers: { accept: 'application/json' }
|
||||
},
|
||||
expect: [
|
||||
'status:200'
|
||||
],
|
||||
capture: {
|
||||
session_cookie: 'response_headers(this)["set-cookie"]'
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'authorize',
|
||||
request: {
|
||||
method: 'GET',
|
||||
url: '/oauth/authorize?...',
|
||||
headers: {
|
||||
accept: 'text/html',
|
||||
cookie: '$login.session_cookie'
|
||||
}
|
||||
},
|
||||
expect: [
|
||||
'status:302',
|
||||
'redirect_count(this) == 1'
|
||||
],
|
||||
capture: {
|
||||
code: 'redirect_query(this).0.code'
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'token',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/oauth/token',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/x-www-form-urlencoded'
|
||||
},
|
||||
form: {
|
||||
grant_type: 'authorization_code',
|
||||
code: '$authorize.code'
|
||||
}
|
||||
},
|
||||
expect: [
|
||||
'status:200',
|
||||
'response_payload(this).access_token != null'
|
||||
],
|
||||
capture: {
|
||||
refresh_token: 'response_payload(this).refresh_token'
|
||||
}
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
This would let APOPHIS test OAuth 2.1, device authorization, WIMSE S2S, transaction tokens, and similar protocol flows in a uniform system.
|
||||
|
||||
### 5. Add First-Class Capture/Rebind Support
|
||||
|
||||
Protocol testing needs more than `previous()`.
|
||||
|
||||
We need first-class support for:
|
||||
|
||||
1. capturing from response body
|
||||
2. capturing from response headers
|
||||
3. capturing from cookies
|
||||
4. capturing from redirect URLs
|
||||
5. rebinding captured values into later request URLs, headers, query, body, and form fields
|
||||
|
||||
This is the difference between route testing and protocol testing.
|
||||
|
||||
Examples:
|
||||
|
||||
1. capture auth code from redirect query
|
||||
2. capture refresh token from token response
|
||||
3. capture session cookie from login response
|
||||
4. capture `request_uri` from PAR response
|
||||
5. reuse all of them in later steps
|
||||
|
||||
### 6. Add a Cookie Jar to Scenario and Stateful Execution
|
||||
|
||||
OAuth and browser-like flows depend on cookies persisting across requests.
|
||||
|
||||
Today APOPHIS can inspect cookies in formulas, but protocol scenarios need an actual cookie jar that automatically:
|
||||
|
||||
1. records `Set-Cookie`
|
||||
2. applies matching cookies on subsequent requests
|
||||
3. can still be overridden explicitly
|
||||
|
||||
Without this, login -> authorize -> consent flows remain awkward and externalized.
|
||||
|
||||
### 7. Add First-Class `application/x-www-form-urlencoded` Request Support
|
||||
|
||||
Token, PAR, revocation, introspection, and device flows rely heavily on form encoding.
|
||||
|
||||
APOPHIS should support request generation and scenario steps with:
|
||||
|
||||
1. `form` bodies
|
||||
2. automatic `content-type: application/x-www-form-urlencoded`
|
||||
3. schema-driven field generation for form posts
|
||||
|
||||
This should be a first-class capability, not a string-construction escape hatch.
|
||||
|
||||
### 8. Add Better Redirect Introspection Helpers
|
||||
|
||||
You already expose redirect count, status, and URL. That is close, but protocol testing needs one more step.
|
||||
|
||||
Suggested additions:
|
||||
|
||||
1. `redirect_query(this).0.code`
|
||||
2. `redirect_query(this).0.state`
|
||||
3. `redirect_fragment(this).0.access_token`
|
||||
|
||||
That would remove a lot of brittle URL parsing from tests.
|
||||
|
||||
### 9. Add Representation and Media-Type Predicates
|
||||
|
||||
Protocol routes often care as much about wire format as about semantic payload.
|
||||
|
||||
Suggested additions:
|
||||
|
||||
1. `response_media_type(this)`
|
||||
2. `request_media_type(this)`
|
||||
3. `representation(this)` returning values like `json`, `ldf`, `html`, `redirect`, `empty`
|
||||
|
||||
This enables formulas like:
|
||||
|
||||
```apostl
|
||||
if request_headers(this).accept matches /application\/ldf\+json/ then representation(this) == "ldf" else true
|
||||
if status:302 then representation(this) == "redirect" else true
|
||||
```
|
||||
|
||||
### 10. Add Protocol Packs Built on Top of the Above
|
||||
|
||||
Once the pieces above exist, APOPHIS could support reusable protocol packs without hardcoding protocol logic into core.
|
||||
|
||||
Examples:
|
||||
|
||||
1. `oauth21ProfilePack()`
|
||||
2. `rfc8628DeviceAuthorizationPack()`
|
||||
3. `rfc8693TokenExchangePack()`
|
||||
|
||||
These packs should be implemented as:
|
||||
|
||||
1. scenario definitions
|
||||
2. invariant bundles
|
||||
3. representation variants
|
||||
4. extension requirements
|
||||
|
||||
That would let applications opt into rich conformance testing without rewriting bespoke harnesses.
|
||||
|
||||
## Suggested Minimal Design
|
||||
|
||||
If you want the smallest possible cut that still unlocks this space, we recommend doing only these first:
|
||||
|
||||
1. `response_payload(this)`
|
||||
2. `contract({ variants: [...] })`
|
||||
3. scenario runner with capture/rebind
|
||||
4. cookie jar in scenarios/stateful tests
|
||||
5. form-urlencoded request support
|
||||
|
||||
Those five changes would already make OAuth 2.1 protocol testing meaningfully tractable.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Without these features, APOPHIS is strongest on CRUD and hypermedia resources, but weak on standards conformance for real protocols.
|
||||
|
||||
That forces teams into a bad tradeoff:
|
||||
|
||||
1. either change production routes to fit APOPHIS better
|
||||
2. or bypass APOPHIS for the most important protocol tests
|
||||
|
||||
The better outcome is:
|
||||
|
||||
1. production routes stay spec-compliant
|
||||
2. APOPHIS understands negotiated representations
|
||||
3. APOPHIS can execute and verify protocol flows directly
|
||||
|
||||
That would make APOPHIS useful not just for application contract testing, but for standards-grade protocol verification.
|
||||
|
||||
## Concrete Arbiter Example
|
||||
|
||||
For Arbiter specifically, this would let us test OAuth routes in the correct way:
|
||||
|
||||
1. `Accept: application/json` -> verify plain RFC responses
|
||||
2. `Accept: application/ldf+json` -> verify LDF/hypermedia responses
|
||||
3. same semantic formulas via `response_payload(this)`
|
||||
4. same route, same handler, same production behavior
|
||||
5. cross-step protocol assertions for authorize -> token -> refresh -> revoke
|
||||
|
||||
That is the capability gap we hit.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
APOPHIS is already close.
|
||||
|
||||
It has most of the primitives. What it lacks is the protocol-testing composition layer.
|
||||
|
||||
If you add:
|
||||
|
||||
1. representation-aware contracts
|
||||
2. semantic payload normalization
|
||||
3. variant execution
|
||||
4. scenario capture/rebind
|
||||
5. cookie jar + form support
|
||||
|
||||
then rich OAuth 2.1 conformance testing becomes something APOPHIS can own directly instead of something that has to live in a separate bespoke harness.
|
||||
Reference in New Issue
Block a user