Day 61 of 100
The Deployment Dependency
🔥 Problem Statement
Service B ships a change — from its own team's perspective, a successful, well-tested deployment. Almost immediately, Service A starts failing. Service A's team didn't deploy anything; their code hasn't changed. What changed is the shape of the API Service A depends on, out from under it, with no warning that this specific deployment would affect anyone downstream.
🏗️ Environment
- Architecture: Service A depends on Service B's API
- Service B: recently deployed a breaking change to its API
- Service A: unchanged, now failing against the new API shape
- No contract testing currently in place between these services
🤔 Your Challenge
How should teams manage compatibility between services that deploy independently but depend on each other's APIs?
- If Service B's own tests all passed, what does that tell you about whether the change was actually safe for its consumers?
- What's the difference between a backward-compatible API change and a breaking one?
- How would Service B's team have known Service A depends on the specific part of the API that changed?
- What would contract testing actually verify that Service B's own tests wouldn't?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
Service B's own tests passing confirms Service B works correctly by its own definition of correct — it says nothing about whether every consumer of its API can still successfully talk to it. A microservices architecture means teams deploy independently, on their own schedules, without necessarily knowing everyone who depends on their API or what specifically those consumers rely on. "Our tests passed" and "we didn't break anyone downstream" are genuinely different claims, and only the first one was actually verified here.
Step 2 — Identify the likely bottleneck
The structural gap is that Service B's team had no visibility into what Service A (or any other consumer) actually depends on in their API's contract — which fields, which behaviors, which response shapes are load-bearing for someone else. Without that visibility, or some automated way to check for breaking changes against real consumer expectations, a change that looks safe from Service B's side alone can break every downstream consumer simultaneously, discovered only after it's already live.
Step 3 — Investigation
- Identify exactly what changed in Service B's API and specifically what part of that change Service A's failure traces back to — a removed field, a changed data type, a modified response structure, a changed status code convention.
- Check whether this change was flagged anywhere as breaking, or whether it was made under the assumption it was additive/backward-compatible when it actually wasn't for at least one real consumer.
- Identify every other consumer of Service B's API, not just Service A — if one consumer broke, it's worth checking whether others are similarly affected even if they haven't reported it yet.
- Check whether any API versioning or contract exists between these services currently, and whether it was respected by this change.
Step 4 — Recommended action
Immediately: either roll back Service B's change or have Service A adapt to the new shape quickly, whichever restores service faster with less risk. Longer term, this points at two complementary fixes: API versioning, so breaking changes can be introduced as a new version while existing consumers continue against the old one until they explicitly migrate; and contract testing, where each consumer's actual expectations of a provider's API are captured as automated tests the provider runs against before deploying — turning "did we break a consumer" from a question discovered in production into one caught in the provider's own CI pipeline.
Progressive rollout (canary or staged deployment) for API-facing services also helps here — a breaking change affecting real consumers would show up as errors in a small percentage of traffic first, rather than affecting every consumer simultaneously the moment it ships.
Step 5 — Engineering lesson
In a distributed system, "my service deployed successfully" and "the system as a whole is still working" are different claims, and independent deployability — one of microservices' core benefits — only stays safe if compatibility between services is actively maintained, not assumed. Contract testing and API versioning exist specifically to make "did this change break someone downstream" a question answered before deployment, rather than one answered by whichever consumer's on-call engineer notices first.
🧠 Today’s Takeaway
Distributed systems require compatibility, not just successful deployments.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

