The Queue That Keeps Growing

Step 1 — Understand the symptoms

A queue's depth is the direct, simple result of one thing: the rate messages are added versus the rate they're removed. If depth is climbing steadily rather than fluctuating around a stable level, that means production rate is exceeding consumption rate, consistently, over the whole observed window — not a momentary burst that a healthy system would absorb and drain, but a sustained mismatch. "Consumers aren't erroring" doesn't mean consumers are keeping up; a consumer can run cleanly, with zero errors, while simply processing too slowly or too few messages per second to match what's coming in.

Step 2 — Identify the likely bottleneck

With a fixed consumer pool and no autoscaling, the most likely explanations split into two categories: producers are genuinely producing more than usual (a real traffic increase, a batch job, an upstream retry storm feeding more messages than normal), or consumers are processing more slowly per message than they used to (a downstream dependency that got slower, a recent code change that added per-message latency, resource contention on the consumer instances themselves). Either way, a fixed-size consumer pool has no way to absorb the mismatch — it just falls further behind, linearly, for as long as the imbalance continues.

Step 3 — Investigation

• Measure actual producer rate and consumer rate directly, not just queue depth — messages published per second versus messages successfully processed per second. This turns "the queue is growing" into an actual number showing how far behind consumption is.
• Check consumer processing latency per message over time — if it's increased compared to a recent baseline, that points at consumers getting slower rather than producers sending more.
• Check for a spike in message volume from producers — a batch job, a retried upstream failure, or a genuine traffic increase would show up here directly.
• Check consumer resource usage (CPU, memory, connection pool) — resource contention on the consumer side is a common, quiet cause of per-message slowdown that doesn't show up as errors.
• Check whether messages are failing and being requeued — a consumer that's processing a message, failing partway, and putting it back on the queue can inflate depth without ever surfacing as a clean "error" in a naive error-rate metric.

Step 4 — Recommended action

If producers are genuinely sending more, the fix is scaling consumers (or making them faster) to match the new sustained rate — a fixed pool sized for yesterday's traffic won't absorb a real, lasting increase. If consumers have gotten slower per message, the fix is finding and addressing that slowdown specifically (a downstream dependency, a resource constraint, a code regression) rather than just adding more consumer instances to compensate for something that's actually broken.

Either way, queue depth deserves its own alert independent of consumer error rate — a queue can grow for hours with zero consumer errors, exactly as it did here, and by the time it's noticed on a dashboard by chance, real backlog and real latency for whatever's waiting in that queue has already accumulated.

Step 5 — Engineering lesson

"No errors" and "keeping up" are different claims about a consumer, and a queue is one of the few places in a system where the gap between them shows up clearly and early — often well before anything else looks wrong. Treating queue depth as a first-class signal, not an afterthought behind error-rate alerting, is what turns "the queue has been silently falling behind for six hours" into "we caught this in the first thirty minutes."