Designing for the failure you can’t see

At 02:14 on a quiet Tuesday, our checkout service began returning errors. The database was healthy, CPU was low, and every dashboard was green. The cause lived in the negative space between services: a downstream request that had stopped failing quickly and started succeeding slowly. Each response arrived just inside its timeout, tying up a worker long enough for the next request to join the queue. Nothing was broken in isolation. Together, the system had become unavailable.

Reliability begins when a system can say “no” before it forgets how to say anything at all.

Mara Chen, Principal Engineer

This is the shape of many production incidents. We tend to model failure as a binary event—a process is up or down, a request succeeds or errors—but distributed systems spend much of their time in less legible states. They become slow, partial, stale, or uneven. A dependency may work for one region and not another, or for small payloads but not large ones. Designing for resilience means treating those ambiguous states as ordinary operating conditions rather than surprises.

The first practical move is to put a budget around every piece of waiting. An incoming request has a finite amount of time in which it remains valuable, and each downstream call spends part of that budget. Pass the deadline through the call graph, reserve time for recovery, and stop work once the result can no longer be used. A timeout should not be an arbitrary constant copied between services; it should be a declaration of how much latency this operation can afford.

async function loadProfile(
  userId: string,
  deadline: number
) {
  const remaining = deadline - Date.now();
  const recoveryReserve = 75;

  if (remaining <= recoveryReserve) {
    return cache.get(userId); // degrade deliberately
  }

  return profiles.fetch(userId, {
    timeout: remaining - recoveryReserve,
    signal: AbortSignal.timeout(remaining - recoveryReserve)
  });
}

Retries need the same discipline. A retry is not free resilience; it is another request arriving precisely when a dependency is least able to serve it. Retry only operations that are safe to repeat, add jitter so clients do not move in lockstep, and cap the total attempt budget rather than the number of attempts alone. When pressure rises, concurrency limits and circuit breakers should shed optional work early. The goal is not to force every request through—it is to preserve enough capacity for the requests that can still succeed.

These controls work best when their effect is visible. Aggregate success rate can conceal the path users actually take, so measure the boundaries: time spent waiting for a connection, queue depth before a worker, deadline remaining at each hop, and the fraction of responses served from a fallback. In one representative load test, a modest concurrency limit increased rejected requests at the edge while sharply reducing end-to-end failures. The system did less work and completed more useful work.

Finally, recovery paths must be exercised before they are needed. A fallback that has not run in six months is not a fallback; it is a hypothesis. Test degraded modes in staging, invoke them safely in production, and make ownership explicit. The deepest lesson from that Tuesday incident was not that we needed a shorter timeout. It was that resilience comes from designing limits as part of the product: clear deadlines, intentional degradation, and enough observability to understand what the system chose not to do.