The Pod That Never Becomes Ready

Step 1 — Understand the symptoms

Running is a container-runtime status. It means the container process started and hasn't crashed. It says nothing about whether the application inside is actually able to serve a request — whether it's finished booting, connected to its dependencies, warmed up its caches, or is otherwise ready to do useful work.

A Kubernetes Service doesn't route traffic to every pod that matches its selector. It routes traffic only to pods listed in the Service's Endpoints — and a pod only gets added to Endpoints once its readiness probe passes. A pod can be Running indefinitely while its readiness probe keeps failing, and from the Service's point of view, that pod simply doesn't exist as a traffic target yet.

Step 2 — Identify the likely bottleneck

The team's assumption — "Running means healthy" — is the actual bug in their reasoning, not the cluster's networking. The far more likely explanation: the new pods are Running, but their readiness probe (GET /healthz) is failing or hasn't started passing yet, so they've never been added to the Service's Endpoints. Traffic keeps flowing to the old ReplicaSet because it's the only one with pods currently marked Ready — and because the rolling update won't scale down old pods faster than new ones become Ready, the rollout is effectively stuck in a holding pattern that looks, at a glance, like a completed deploy.

Common root causes for a Running-but-never-Ready pod: the new version's /healthz endpoint checks a dependency (database, cache, a downstream service) that isn't reachable from this environment yet; a longer startup time than the probe's initialDelaySeconds accounts for, so the probe starts checking before the app has finished booting; or the health endpoint itself changed in this release and no longer returns what the probe expects.

Step 3 — Investigation

Before assuming a networking problem, the pod's own status has the answer:

kubectl describe pod <new-pod-name>

Look specifically at the Conditions section for Ready: False, and at Events near the bottom — a failing readiness probe shows up there explicitly, often with the HTTP status code or timeout it received.

kubectl get endpoints <service-name>

Confirms directly whether the new pods are in the Endpoints list at all — if they're missing, that's conclusive: the Service was never going to send them traffic, independent of anything at the networking layer.

kubectl logs <new-pod-name>

Checks whether the application logged a real error on startup — a failed dependency connection, a config value that didn't get set, anything explaining why /healthz would be failing.

Step 4 — Recommended action

Fix the readiness signal, not the network. If the probe is failing because a dependency isn't reachable yet, that's either a real environment issue (fix the dependency) or a probe that's too aggressive for a legitimate startup sequence (adjust initialDelaySeconds or add a proper startup probe). If the health endpoint's behavior changed in this release, that's a code fix, and the rollout should be rolled back in the meantime rather than left half-finished — a stuck rollout with old and new ReplicaSets both present is its own risk, not a stable state to leave running.

Don't touch the Ingress or Service configuration on a hunch. If kubectl get endpoints shows the new pods genuinely missing, the networking layer is doing exactly what it's supposed to do with the information it has — the problem is upstream of it, in the pod's own readiness signal.

Step 5 — Engineering lesson

Running and Ready answer two completely different questions, and conflating them is one of the most common ways a "clean" deploy turns into a quiet, half-finished rollout. Kubernetes is already telling you which one matters for traffic — the Endpoints object is the ground truth for "can this pod actually receive requests right now," and it's one command away.

If an on-call runbook says "check pod status" and stops there, it's missing the step that actually explains most silent-traffic incidents: check readiness, not just liveness.