Day 46 of 100
The Autoscaling Oscillation
🔥 Problem Statement
Instance count won't sit still. It scales up, then a few minutes later scales back down, then up again — a repeating cycle rather than a stable response to steady load. Every cycle brings a noticeable dip in performance as new instances come up and existing ones get removed, right in the middle of otherwise normal traffic.
🏗️ Environment
- Autoscaler: Horizontal Pod Autoscaler, CPU-based metric
- Scaling behavior: repeatedly scales up then down in short cycles
- Cooldown/stabilization: default configuration, not tuned
- Symptom: performance fluctuates in sync with each scaling cycle
🤔 Your Challenge
Why might an autoscaler oscillate instead of settling into a stable instance count, and what would you tune?
- What happens if new instances take a while to become fully useful, but the scaling metric reacts immediately?
- What role does a cooldown or stabilization window play in preventing rapid back-and-forth scaling?
- Could the metric itself be noisy or bursty in a way that a smoother metric wouldn't be?
- What's the difference between fast scaling and stable scaling?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
Oscillation — scale up, then down, then up again, on a short cycle — usually means the autoscaler is reacting to signal changes faster than the system can actually stabilize in response to its own scaling actions. If new instances take meaningful time to become fully warmed up and contributing capacity, but the scaling metric (like average CPU) reacts to load in real time, the autoscaler can see elevated CPU, add instances, watch CPU drop as those instances start absorbing load, then scale back down — sometimes before the added instances were ever fully useful, restarting the cycle.
Step 2 — Identify the likely bottleneck
Default cooldown/stabilization settings are often too short for this specific interaction, especially for workloads where new instances take real time to warm up (JIT-compiled runtimes, connection pool establishment, cache population) before they're contributing full capacity. Without a long enough stabilization window, the autoscaler evaluates and reacts to metric changes faster than the true, settled effect of its last scaling action has had time to show up — creating exactly this kind of feedback loop.
Step 3 — Investigation
- Check the autoscaler's stabilization window configuration for both scale-up and scale-down — many autoscalers default to fairly short windows that work fine for simple workloads but not ones with real startup latency.
- Measure actual time-to-ready for new instances — from creation to fully contributing capacity, not just "container running." If this is a meaningful fraction of the scaling cycle's length, it's very likely the core cause.
- Check the smoothness of the underlying metric — is CPU usage itself noisy or bursty in a way that produces frequent threshold crossings even under genuinely stable load? A noisy signal feeding a fast-reacting autoscaler compounds the oscillation risk.
- Check the specific scale-up and scale-down thresholds — if they're close together, normal metric variance can cross both repeatedly; wider separation (hysteresis) between them reduces sensitivity to minor fluctuations.
Step 4 — Recommended action
Lengthen the stabilization window, particularly for scale-down, so the autoscaler waits long enough after a scaling action to observe its true, settled effect before making another decision — this directly breaks the feedback loop causing rapid cycling. If instance warm-up time is significant, consider whether readiness gating is correctly configured so new instances aren't counted as contributing capacity (and therefore don't trigger premature scale-down) until they're actually ready to serve real traffic.
If the underlying metric itself is noisy, consider a metric with more inherent smoothing (a moving average, or a different signal like request concurrency rather than instantaneous CPU) as the scaling trigger, and widen the gap between scale-up and scale-down thresholds to add hysteresis against normal variance.
Step 5 — Engineering lesson
An autoscaler reacting quickly isn't automatically a good thing — reacting quickly to a signal that hasn't yet reflected the true, settled state of the system just means reacting to noise, over and over. Stable autoscaling requires giving the system enough time between actions to actually show the effect of the last one before deciding on the next — speed and stability are different goals, and tuning purely for the first can actively work against the second.
🧠 Today’s Takeaway
Fast scaling isn't always stable scaling.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

