Imagine a late-night incident: a database is blocked, an autoscaling policy increases load or cost, and the on-call team ships a fix. That recovery is visible and often praised. This is an illustrative scenario, not a claim about an incident the author or reader actually experienced.
But what about the system that didn't crash? What about the migration that went so smoothly that nobody even noticed it happened?
Preventative work can be difficult to recognize because its intended outcome is a failure that does not occur. Nelson Repenning and John Sterman examined this problem in their paper published in July 2001, not 2002. See the original publication record. Reliable operation does not mean literally nothing happens; it means desired service continues within defined limits.
Today, we’re going to look at the engineering discipline behind preventative design. How do we write code and architect systems that quietly neutralize catastrophes before they start? More importantly, how do we, as developers, quantify and demonstrate the value of this invisible work so we don't get overlooked when review season rolls around? Let’s dive in.
The Physics of Software Failure
A useful reliability model assumes dependencies can fail or slow down and asks what the system will do. Experience level does not divide neatly into developers who ignore failure and engineers who anticipate it. The important practices are explicit assumptions, timeouts, resource limits, safe retries and recovery tests.
To prevent disasters silently, we rely on three core architectural patterns: Idempotency, Circuit Breaking, and Graceful Degradation. Let's look at how to implement these in real-world code.
1. Designing for Silent Recovery: Idempotent APIs
One of the most common production disasters is the "double-spend" or duplicate action problem. A user clicks "Buy Now," the payment gateway processes the payment, but the network drops before the client gets the 200 OK. The user clicks "Buy Now" again. If your API isn't idempotent, you've just double-charged your customer and triggered a manual reconciliation nightmare for your finance team.
Idempotency means repeated requests for the same intended operation do not repeat its intended effect. It requires an explicit request identity and a durable outcome policy, not merely a short-lived mutex. The original Redis middleware is unsafe as a payment guarantee: it permits missing keys, does not bind keys to caller/operation/payload, uses a fixed lock value, deletes locks without checking ownership and saves results asynchronously after the effect. A crash or expired lease can leave a repeatable action. See AWS on idempotent APIs and Redis lock ownership guidance.
# Architecture pseudocode, not runnable middleware or a payment integration.
# Scope: the business mutation and idempotency record share one durable database.
# A unique constraint covers (authenticated_caller, operation, idempotency_key).
validate authorization, key format, request size and business parameters
scope = (authenticated_caller, operation, idempotency_key)
fingerprint = canonical_digest(validated_business_parameters)
begin database transaction
atomically insert a pending request record for scope, or lock its existing row
if a record already exists:
if its fingerprint differs: reject reuse of the key
if it is complete: return its saved outcome after ending this transaction
otherwise: apply a documented wait/retry/recovery policy
else:
apply the business mutation within this same transaction
persist the outcome in the request record
commit transaction
return the persisted outcome
# For an external effect, such as a payment provider, one local transaction
# cannot make the remote call atomic. Use an outbox/workflow, a stable provider
# idempotency key, provider-status reconciliation and explicit recovery states.
# Define retention, retries, error outcomes and operator recovery before release.
A correctly designed idempotency contract reduces duplicate effects under specified conditions. It must be tested for simultaneous requests, process crashes, timeouts, changed payloads, stale keys and datastore failures. Response caching alone does not establish exactly-once execution, and no middleware guarantees zero incident pages.
2. The Circuit Breaker: Failing Fast to Stay Alive
A slow dependency can consume concurrency, connections or queue capacity and contribute to cascading failure. The outcome depends on isolation and limits; a slow service does not inevitably bring down every other service. Google’s SRE discussion explains why retries, overload and failover interactions deserve explicit testing.
A circuit breaker can stop new calls after a failure threshold is reached. Combine it with deadlines, concurrency limits and bounded retry policies. A fallback must be semantically safe: stale product descriptions may be acceptable, but silently bypassing authentication or claiming a successful payment is not.
The Architecture of a Circuit Breaker
Think of it as a state machine with three states:
- Closed: requests are allowed while results are monitored; this state does not prove that every dependency is healthy.
- Open: calls are rejected for a controlled interval or sent to an explicitly safe alternative.
- Half-Open: After a cooldown, allow a few test requests to see if the service recovered.
Reducing calls can help a dependency recover, but the fallback path must also have capacity limits. Otherwise the fallback itself can become a new bottleneck or conceal a long-lived failure.
How to Get "Credit" for the Disasters You Prevented
Now, let's address the elephant in the room. If your code is elegant, resilient, and never crashes, how do you prove your value during performance reviews? In a corporate culture that rewards firefighting, how does the fire preventer survive?
You have to change the metric. You have to make the absence of problems visible and quantifiable. Here is how you do it.
1. Instrument Your Resiliency Patterns
Measure breaker transitions, rejected calls, latency, fallback success and user-visible outcomes. A graph shows what happened, not proof that a disaster would otherwise have occurred. Use bounded labels and avoid personal data; Prometheus naming guidance warns against unbounded cardinality.
For example, count a replay only after the durable request record confirms a completed operation. Use a route template rather than a raw path containing user identifiers:
// Conceptual instrumentation; adapt to the chosen metrics library.
if (completedRequestWasReplayed) {
metrics.increment("api_idempotency_replays_total", {
operation: "create_order" // bounded name, not a user ID or raw request path
});
}
A useful review statement separates observation from estimate: “We observed X replayed requests during this period. Based on documented support handling assumptions, the estimated avoided work is Y hours.” The original figures of 14,200 requests and 40 hours were examples, not measured results; do not present them as personal achievements.
2. Introduce Failure injection (Chaos Engineering)
Failure injection requires a defined hypothesis, authorized scope, monitoring, stop conditions and rollback. Start in an appropriate test environment. Production experiments require explicit operational authorization and a bounded blast radius, not merely courage. See the principles of chaos engineering. No failure injection is performed here.
Record the experiment, load, duration and user-facing success/latency measures. The source’s 95% checkout figure was hypothetical and “efficiency” was undefined. A successful experiment increases confidence within its conditions; it does not prove resilience to all failures or establish an absolute business value.
3. Frame Your Work in "Risk Mitigation"
Product managers speak in features; business leaders speak in risk and revenue. When proposing architectural work like writing tests, refactoring legacy code, or updating dependencies, stop framing it as "technical debt cleanup."
Explain the affected service, plausible failure mode, exposure period and impact estimate. If you cite an outage cost, show the underlying data and uncertainty. The source’s 25,000 dollars per minute was an illustrative number, not a verified historical cost for this organization.
Conclusion: The Ultimate Engineering Flex
Incident response and preventative engineering are both valuable. A healthy engineering culture recognizes tested safeguards, maintained recovery plans and useful instrumentation as well as effective emergency response.
When you add a fallback, timeout or test, record which failure mode it addresses and how you checked it. The benefit may be real without being directly measurable, but uncertainty should remain explicit rather than being converted into invented savings.
Make preventative work visible through measured outcomes, documented assumptions and authorized tests. That creates a stronger case for maintenance than promising that failures will never happen.
What do you think?
How do you demonstrate the value of your invisible engineering work at your current job? Have you ever successfully negotiated for "maintenance time" by framing it as risk mitigation? Let’s talk about it in the comments below!