Skip to content

Day 27 of 100

The Ingress 502 Mystery

KubernetesNetworkingIntermediate15–20 min

🔥 Problem Statement

Users occasionally get HTTP 502 errors hitting the public endpoint — not constantly, intermittently, maybe one request in twenty. Testing the pods directly from inside the cluster, bypassing the public path entirely, shows them responding correctly every time.

The failure is happening somewhere between the client and the pod, but "somewhere in a multi-hop path" isn't a place to actually start fixing anything.

🏗️ Environment

  • Path: Client -> DNS -> Load Balancer -> Ingress -> Service -> Pod
  • Ingress controller: managed, standard configuration
  • Pods: healthy per direct in-cluster testing
  • Symptom: intermittent HTTP 502 from the public endpoint

🤔 Your Challenge

Given the full hop-by-hop path from client to pod, how would you narrow down which specific hop is producing the 502?

  • What does a 502 specifically mean, and which layer typically generates that status code versus passes it through?
  • If pods are healthy when tested directly, what does that rule out and what does it leave in scope?
  • How would you test each hop — load balancer, ingress, service — independently rather than only end-to-end?
  • Why would this be intermittent rather than constant, if one specific hop is the cause?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

A 502 (Bad Gateway) specifically means something acting as a proxy — the load balancer, or the ingress controller — received an invalid or no response from whatever it was forwarding to, and is reporting that failure on its own behalf. That's a meaningfully different signal than a 500 from the application itself; a 502 usually means the problem is at or before the proxy layer, not inside the application that never actually got to respond.

Step 2 — Identify the likely bottleneck

Pods responding correctly to direct in-cluster testing rules out the application itself as the source. That narrows the scope to the hops between the public entry point and the pod: the load balancer's health checking and connection handling, the ingress controller's own health and configuration, or the Service in between. An intermittent (not constant) 502 fits a few specific patterns well: a subset of pods being intermittently unready and briefly removed from rotation, ingress controller connection timeouts that are slightly too aggressive for occasional slow responses, or backend connection pool exhaustion at the ingress layer under moderate concurrent load.

Step 3 — Investigation

Test each hop in isolation rather than only end-to-end:

  • Confirm pod health and readiness directly — already done, and worth re-confirming specifically during a window when 502s are occurring, not just generally.
  • Check ingress controller logs for the specific requests that received a 502 — most ingress controllers log which upstream failed and why (timeout, connection refused, connection reset), which usually points precisely at the cause.
  • Check the Service's endpoint list over time, not just once — if pods are flapping in and out of readiness intermittently, the endpoint list itself would show pods appearing and disappearing, correlating with when 502s occur.
  • Check ingress controller timeout and connection-pool settings against actual backend response time under moderate load — a timeout set close to typical response time will intermittently trip on normal variance.
  • Check load balancer health-check configuration separately from ingress-level checks — a load balancer marking backend ingress instances unhealthy intermittently is a distinct possible cause from an ingress-to-pod issue.

Once the specific hop is identified via logs and endpoint history, the fix follows directly: if pods are flapping readiness, address whatever's causing that (often a readiness probe that's too sensitive, similar to a prior day's lesson); if ingress timeouts are too tight, widen them to a realistic margin; if connection pools are being exhausted, increase capacity or investigate why connections aren't being released promptly.

Resist fixing this by retrying at the client or adding a blanket "just increase all the timeouts everywhere" change before the logs actually point at a specific hop — that treats the symptom without confirming the cause, and risks masking the real issue rather than resolving it.

Step 5 — Engineering lesson

A request that fails somewhere in a five-hop path (client → DNS → load balancer → ingress → Service → pod) can't be debugged as one thing — it has to be tested and reasoned about hop by hop, ruling each one in or out with actual evidence. "It's intermittent and pods are healthy" narrows the search space usefully but doesn't answer the question on its own; the ingress controller's own logs, specifically, are usually the fastest path from "somewhere in the middle" to an actual named cause.

🧠 Today’s Takeaway

Troubleshoot distributed systems hop by hop.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.