- Chaos testing injects deliberate failure — latency, dropped connections, 5xx responses, rate-limit rejections — to verify a system degrades gracefully instead of collapsing.
- It answers a different question from load testing: load asks whether you survive volume, chaos asks whether you survive faults.
- Every experiment should start from a written hypothesis. Without one you generate noise rather than evidence.
- Most of the value is available in staging, at a fraction of the risk of production experiments.
What is chaos testing?
Chaos testing — often called chaos engineering — deliberately introduces faults into a running system to observe how it responds. The premise is straightforward: these conditions will occur in production eventually, so it is better to trigger them on a Tuesday afternoon with the team watching than at 3am with nobody sure what changed.
For APIs specifically, it means intercepting calls to dependencies and making them behave badly: adding two seconds of latency, returning a 503, closing the connection mid-response, or rejecting with a 429. Then you check whether the timeouts, retries and circuit breakers you wrote actually do what you believe they do.
Almost every team has resilience code. Very few have evidence it works. A retry policy that has never been exercised is a hypothesis, not a safeguard — and a surprising proportion of them make outages worse by amplifying load against an already-struggling dependency.
Chaos testing vs load testing
These are frequently confused, and they find entirely different defects. A service can sustain ten thousand requests per second and still fall over completely when one downstream dependency starts responding in 30 seconds instead of 30 milliseconds — because every worker thread ends up blocked waiting on it.
| Load testing | Chaos testing | |
|---|---|---|
| Question | Does it survive volume? | Does it survive faults? |
| Variable | Traffic rate | Dependency behaviour |
| Typical finding | Saturation point, slow queries | Missing timeouts, retry storms, cascading failure |
| Verifies | Capacity | Circuit breakers, fallbacks, graceful degradation |
The fault types worth injecting
Latency injection
Add delay to a dependency's responses. This is the highest-value single experiment, because latency is the failure mode most systems handle worst. A dependency that is slow rather than down keeps connections occupied, and without an aggressive timeout the calling service exhausts its own thread or connection pool — turning one slow dependency into a total outage.
Error injection
Return 500 or 503 from a dependency at a controlled rate. Verify the caller distinguishes retryable from non-retryable failures, applies exponential backoff with jitter, and gives up rather than retrying indefinitely.
Connection failure
Refuse connections or close them mid-response. Truncated responses are especially revealing: code that parses a partial JSON body often throws an unhandled exception rather than treating it as a failed call.
Rate-limit simulation
Return 429 with a Retry-After header. Confirm the client honours the header rather than immediately retrying — a client that ignores Retry-After under load will keep a rate-limited dependency permanently saturated.
Running an experiment properly
- State the hypothesis in writing. "If the payments service adds two seconds of latency, checkout still completes because the circuit breaker opens after five consecutive failures and we fall back to the queued path." A hypothesis you can be wrong about is the entire point.
- Define the blast radius. Which service, which percentage of traffic, which environment. Start small enough that being wrong is cheap.
- Define the abort condition. The specific metric and threshold at which you stop — and confirm you can stop in a single action before you start.
- Measure the steady state first. You cannot detect degradation without a baseline taken minutes earlier, not last quarter.
- Inject the fault and observe. Watch the caller, not just the faulted dependency. Cascading failure is the finding that matters.
- Write down what actually happened. A disproved hypothesis is the most valuable outcome — it is a real defect found without an incident.
What chaos testing typically uncovers
- Missing or absurdly generous timeouts. Default HTTP client timeouts are frequently 30 seconds or unlimited. Under latency injection, this is the fault that cascades.
- Retry storms. Three services each retrying three times turns one failed request into 27, applying maximum load exactly when a dependency is least able to serve it.
- Circuit breakers that never open. Thresholds set so high the breaker is decorative.
- Fallbacks that fail too. A cache fallback that calls the same unavailable service to populate itself.
- Error handling that leaks. Dependency failures surfacing to users as stack traces or raw 500s rather than a handled message.
Is it safe to run in production?
With discipline, yes — that is where the highest-fidelity signal lives, since staging never quite matches production's data volume, traffic mix or infrastructure. But the sequence matters: prove the experiment in staging first, scope it to a small percentage of traffic, run it during business hours with the team available, and make sure a single action stops it.
For most teams the honest answer is that staging experiments deliver the majority of the value at a small fraction of the risk. Production chaos is worth graduating to, not starting with.
Test your API without writing the tests
Paste a URL. Flasqo discovers your endpoints, generates the suite and runs it — free, no credit card.
Start testing free