Day 47 of 100
The Noisy Neighbor
🔥 Problem Statement
Every time a particular batch job runs, unrelated services elsewhere in the cluster start experiencing degraded performance — increased latency, occasional throttling — even though nothing about those other services changed. The batch job itself completes successfully every time; from its own team's perspective, everything's fine. It's everyone sharing the same nodes who notices the impact.
🏗️ Environment
- Cluster: shared, multiple teams' workloads on the same node pool
- One workload: batch job with no CPU limit set
- Other workloads: normal resource requests/limits, sharing the same nodes
- Symptom: unrelated services degrade whenever the batch job runs
🤔 Your Challenge
How should a platform team design resource isolation so one workload can't degrade others sharing the same infrastructure?
- What happens to other workloads on a node if one pod has no CPU limit and consumes as much as it can get?
- What's the difference between a resource request and a resource limit, and what does the absence of a limit actually allow?
- Could this workload be moved to different scheduling boundaries entirely, rather than just tuning its limits?
- What platform-level guardrails would prevent this pattern from recurring with a different workload later?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
A pod with no CPU limit set can consume as much CPU as is available on its node whenever it's actively running — there's no ceiling constraining it. If other pods on the same node have requests but the node's actual available CPU gets consumed by the unbounded batch job, those other pods can be squeezed for CPU time even if their own configuration looks perfectly reasonable in isolation. This is the classic "noisy neighbor" problem: one tenant's workload affecting others purely through shared infrastructure contention, with no code-level bug in either workload.
Step 2 — Identify the likely bottleneck
The batch job having no CPU limit is the direct, specific cause — it's allowed to consume unbounded CPU on whatever node it lands on, and Kubernetes' own scheduling and fair-share mechanisms only go so far in protecting other pods when one pod is actively, legitimately requesting far more CPU than its neighbors. This isn't a bug in the batch job; it's an absence of a boundary that should exist at either the workload level or the platform level.
Step 3 — Investigation
- Confirm the batch job's resource configuration — presence or absence of CPU/memory requests and limits, and how they compare to what it actually consumes when running.
- Check which other workloads share nodes with this batch job, and whether the degradation reported correlates specifically with the batch job's run windows.
- Check the node pool's overall capacity versus what's being requested/consumed during the batch job's execution — this quantifies how much genuine contention exists versus available headroom.
- Check whether resource quotas or LimitRanges exist at the namespace level for this workload — their absence is often the platform-level gap that allowed an unbounded pod to be scheduled onto shared infrastructure in the first place.
Step 4 — Recommended action
Set an explicit CPU limit on the batch job, sized to what it actually needs to complete in a reasonable time without being able to consume unbounded node capacity. Beyond fixing this one workload, this is a good moment to establish platform-level guardrails: LimitRanges or default resource limits at the namespace level so a future workload can't be scheduled without bounds either, and resource quotas per namespace or team to cap total consumption regardless of any individual workload's configuration.
For genuinely resource-intensive batch or background work, consider a separate node pool — one dedicated to batch workloads, isolated from latency-sensitive services — so contention, if it happens, stays contained to workloads that can tolerate it rather than affecting user-facing services sharing the same nodes.
Step 5 — Engineering lesson
Shared infrastructure means one workload's resource behavior can affect every other workload on the same node, and the absence of an explicit limit isn't neutral — it's implicitly a claim on however much capacity happens to be available, at whoever else's expense that comes from. Multi-tenant platforms need isolation boundaries enforced structurally (limits, quotas, dedicated node pools), not left to each team's individual discipline about setting reasonable resource configuration on their own.
🧠 Today’s Takeaway
Shared infrastructure needs isolation boundaries.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

