Skip to content

Day 20 of 100

The Slow External API

ObservabilityDistributed SystemsIntermediate15–20 min

🔥 Problem Statement

API latency creeps up, and every internal metric says everything's fine — application CPU, memory, and database load are all sitting exactly where they normally do. Nothing internal explains the slowdown.

A trace on one of the slow requests answers the question immediately: most of the elapsed time isn't inside your own code at all. It's spent waiting on a third-party external API your service calls partway through handling the request.

🏗️ Environment

  • Application: healthy, normal CPU/memory/database metrics
  • Dependency: third-party external API, no SLA guarantees
  • Observability: distributed tracing available
  • Symptom: elevated P95 latency on requests that call the external API

🤔 Your Challenge

Given that the third-party API is outside your control, how should the system actually be designed to handle it being slow or unreliable?

  • If the external API just gets slower sometimes, what happens to your own service's latency and capacity by default, with no extra handling?
  • What's the difference between retrying a slow call and giving up gracefully?
  • What happens to your system if every request piles up waiting on the same slow dependency simultaneously?
  • Is there a reasonable fallback if the external API is unavailable, or does the whole request have to fail?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

Normal internal metrics with elevated overall latency is exactly the shape of a problem living outside the service's own boundary. The trace makes this concrete rather than inferred — it shows precisely how much of the request's total time is spent inside the external API call specifically, versus everywhere else. When that single span dominates the trace, the conclusion isn't "our service is slow," it's "our service is only as fast as the slowest thing it waits on."

Step 2 — Identify the likely bottleneck

The bottleneck itself isn't really the question here — it's confirmed, it's the external API. The actual engineering problem is that, without deliberate handling, a slow or unreliable external dependency becomes a slow or unreliable version of your service. Every request that calls it inherits its latency directly; if it's down entirely, your service can end up hanging on every request that touches it, potentially exhausting connections or threads waiting on a dependency that was never yours to guarantee.

Step 3 — Investigation

Rather than digging further into a root cause that's already found, the investigation here is really about understanding the failure's shape well enough to design around it:

  • Check how often this dependency is slow versus consistently slow — an occasional spike calls for different handling (timeout + retry) than a dependency that's degraded for extended periods (circuit breaker).
  • Check what happens today when the external API times out or errors — does the request just fail, hang indefinitely, or retry blindly? The current behavior under failure is often worse than assumed until actually traced.
  • Check whether every request that touches this dependency shares a connection pool or thread pool — if so, a slow external API can exhaust that shared resource and start affecting requests that don't even call it.
  • Check if there's a reasonable fallback or degraded response for this specific call — sometimes stale cached data or a partial response is better than a hard failure; sometimes there genuinely isn't a fallback and the honest answer is the request has to fail.

Add an explicit timeout on the call to the external API — waiting indefinitely for someone else's system is never the right default. Add retries with exponential backoff and jitter for transient failures, but cap them; retrying forever just delays an inevitable failure while holding resources. If the dependency degrades for sustained periods, a circuit breaker — stopping new calls for a cooldown period after repeated failures — prevents every request from individually discovering and waiting out the same outage. If there's a reasonable fallback (cached data, a degraded response, skipping an optional enhancement), use it rather than failing the whole request over one non-critical dependency.

None of this fixes the external API — that's not possible from your side. It bounds the damage it can do to your own system when it's inevitably slow or unavailable, which is the actual goal.

Step 5 — Engineering lesson

Every external dependency you call becomes part of your system's actual reliability, whether or not you control it. The question worth asking isn't "how do we make the third-party API faster" — you usually can't — it's "what does our system do when it's slow, and is that an acceptable answer." Timeouts, retries with limits, circuit breakers, and fallbacks are the difference between a slow dependency degrading your service gracefully and a slow dependency taking it down entirely.

🧠 Today’s Takeaway

Your reliability is partly determined by the dependencies you don't control.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.