Skip to content

Day 21 of 100

The Retry Storm

SREDistributed SystemsIntermediate15–20 min

🔥 Problem Statement

One service in a chain starts responding slowly — not failing outright, just slower than its callers' timeouts expect. Each caller, configured with a fixed number of retries and no backoff, retries almost immediately on timeout. Their callers do the same. Within minutes, the slow service is receiving several times its normal request volume, entirely made up of retries stacked on retries.

What started as one service having a bad moment has become every service in the chain hammering it simultaneously, and its real capacity is now buried under repeated attempts at requests that were already in flight.

🏗️ Environment

  • Architecture: chain of services, each calling the next with default retry logic
  • Retry config: fixed retries, no backoff, no jitter
  • Trigger: one downstream service becomes slow, not down
  • Symptom: overall traffic to the slow service multiplies rapidly

🤔 Your Challenge

How can retries — a mechanism meant to improve reliability — end up amplifying a small slowdown into a large outage?

  • If a service is slow but not down, what happens to total load on it when every caller retries immediately on timeout?
  • What's the difference between one caller retrying and every service in a multi-hop chain retrying independently?
  • Why does retrying without backoff make a slow service's situation worse rather than better?
  • What would 'jitter' actually change about how retries land on the struggling service?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

The service didn't fail — it slowed down. That distinction matters enormously for what happens next: a service that's down fails fast, and callers get a clear, quick error. A service that's slow keeps accepting requests, just too slowly, which means every caller waits out its timeout, gets nothing, and — with retry logic configured — immediately tries again. Multiply that by every service in the chain doing the same thing independently, and the slow service's incoming load doesn't just stay elevated, it compounds with every retry layer added upstream.

Step 2 — Identify the likely bottleneck

The retry configuration itself is the amplifier here: fixed retries with no backoff means every failed attempt is followed immediately by another attempt, with no pause to let the struggling service recover or drain its existing backlog. And with retries happening at multiple hops in the chain — not just the immediate caller, but that caller's caller too — the effective multiplication is worse than any single layer's retry count alone; it compounds across every hop that has its own independent retry logic.

Step 3 — Investigation

  • Trace the actual request multiplication factor — how many total attempts is one original request generating by the time it's retried at every hop in the chain? This turns "traffic multiplied" into a concrete number worth fixing.
  • Check each service's retry configuration individually — retry count, backoff strategy (or lack of one), and whether jitter is applied. This is usually where the actual misconfiguration lives.
  • Check whether the originally-slow service's real capacity is being consumed by genuinely new requests or by repeated retries of requests that are, from its perspective, already queued and being worked on — if it's mostly retries, the service may have had enough capacity all along, just none of it visible to callers who gave up too early.
  • Check for any circuit breakers currently in place — their absence is often exactly what allows a retry storm to develop instead of being cut off early.

Add exponential backoff to every retry layer — each subsequent retry waits longer than the last, giving the struggling service room to recover instead of being hit immediately and repeatedly. Add jitter — randomizing the exact wait time — so that retries from many callers don't land in synchronized waves that create their own periodic spikes even with backoff in place. Cap the total number of retries, and consider a circuit breaker at each hop so that after a run of failures, callers stop retrying for a cooldown window instead of continuing to add load to an already-struggling service.

This is a multi-service configuration change, not a single fix — a retry storm caused by every hop in a chain needs every hop's retry behavior addressed, not just the one closest to the original slowdown.

Step 5 — Engineering lesson

Retries are a reliability tool that becomes a reliability hazard the moment they're uncoordinated — no backoff, no jitter, no cap, across multiple hops that don't know about each other's retry behavior. A single slow service having a bad five minutes shouldn't be able to turn into an outage; the fact that it can here is a design gap in how failure is handled across the whole chain, not a flaw in any one service.

🧠 Today’s Takeaway

Retries can protect a system—or destroy it.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.