John Dvorak
8d7382417d
- Cite arxiv 2602.23922 (Invariant-Driven Automated Testing) in all major docs - Add Progressive Complexity section to SKILL.md for LLM guidance - Fix SKILL.md Fast Start example to use deterministic ID generation - Fix getting-started.md failure output inconsistency - Fix auth-patterns.md TypeScript syntax in JS doc - Fix fastify-structure.md Date.now() in test helper - Fix observe.md misleading workspace heading - Build: clean | Tests: 849 pass, 0 fail
146 lines
3.4 KiB
Markdown
146 lines
3.4 KiB
Markdown
# Chaos Mode
|
|
|
|
Inject controlled failures into contract tests to validate resilience guarantees.
|
|
|
|
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
|
|
|
## Usage
|
|
|
|
```javascript
|
|
const result = await fastify.apophis.contract({
|
|
depth: 'standard',
|
|
chaos: {
|
|
probability: 0.1, // 10% of requests get chaos
|
|
delay: { probability: 1, minMs: 100, maxMs: 500 },
|
|
error: { probability: 1, statusCode: 503 },
|
|
dropout: { probability: 1 },
|
|
corruption: { probability: 1 },
|
|
},
|
|
});
|
|
```
|
|
|
|
## Event Types
|
|
|
|
### Delay
|
|
|
|
Adds artificial latency. Tests timeout contracts:
|
|
|
|
```apostl
|
|
timeout_occurred(this) == false
|
|
response_time(this) < 1000
|
|
```
|
|
|
|
### Error
|
|
|
|
Forces HTTP status codes. Tests error-handling contracts:
|
|
|
|
```apostl
|
|
if status:503 then response_body(this).retry_after != null
|
|
```
|
|
|
|
### Dropout
|
|
|
|
Simulates network failure (status 0). Tests fallback contracts:
|
|
|
|
```apostl
|
|
status:200 || status:0
|
|
```
|
|
|
|
### Corruption
|
|
|
|
Mutates response bodies. Tests parsing robustness:
|
|
|
|
```apostl
|
|
response_body(this).id != null
|
|
```
|
|
|
|
## Corruption Strategies
|
|
|
|
Built-in strategies are content-type agnostic:
|
|
|
|
| Strategy | Effect |
|
|
|----------|--------|
|
|
| `truncate` | Cuts response body short |
|
|
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
|
|
| `field-corrupt` | Replaces a random field value with corrupted data |
|
|
|
|
Extension strategies can add content-type-specific behavior if needed.
|
|
|
|
## Custom Corruption via Extensions
|
|
|
|
```javascript
|
|
const myExtension = {
|
|
name: 'custom-corrupt',
|
|
corruptionStrategies: {
|
|
'application/vnd.api+json': (data) => ({
|
|
...data,
|
|
corrupted: true,
|
|
}),
|
|
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
|
},
|
|
};
|
|
|
|
await fastify.register(apophis, {
|
|
extensions: [myExtension],
|
|
});
|
|
```
|
|
|
|
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
|
|
|
## Environment Guard
|
|
|
|
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
|
|
|
```
|
|
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
|
|
```
|
|
|
|
## Interpreting Results
|
|
|
|
Failed tests include chaos events in diagnostics:
|
|
|
|
```json
|
|
{
|
|
"statusCode": 503,
|
|
"error": "Contract violation: status:200",
|
|
"chaosEvents": [
|
|
{
|
|
"type": "error",
|
|
"injected": true,
|
|
"details": {
|
|
"statusCode": 503,
|
|
"reason": "Chaos error: overridden 200 with 503"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Start small**: `probability: 0.05` (5% of requests)
|
|
2. **Test one failure mode at a time**: Comment out other chaos types
|
|
3. **Verify contracts handle chaos**: `if status:503 then response_body(this).error != null`
|
|
4. **Use seeds for reproducibility**: `seed: 42` makes chaos deterministic
|
|
|
|
## Example: Testing Retry Logic
|
|
|
|
```javascript
|
|
fastify.get('/data', {
|
|
schema: {
|
|
'x-ensures': [
|
|
'if status:503 then response_headers(this).retry-after != null',
|
|
'redirect_count(this) <= 3',
|
|
],
|
|
},
|
|
}, handler);
|
|
|
|
// Test
|
|
const result = await fastify.apophis.contract({
|
|
chaos: {
|
|
probability: 0.2,
|
|
error: { probability: 1, statusCode: 503 },
|
|
},
|
|
});
|
|
```
|