Day 41 of 100
The Noisy Alert
🔥 Problem Statement
An alert fires around 200 times a day. Nobody investigates it anymore — the pattern is well established at this point: it fires, someone acknowledges it, it resolves itself within a few minutes, life goes on. It's been this way long enough that dismissing it has become an unconscious habit rather than a decision anyone actively makes.
🏗️ Environment
- Alert: fires roughly 200 times per day
- Response pattern: routinely acknowledged and dismissed without action
- Underlying condition: usually resolves on its own within minutes
- No SLO-based threshold currently in use for this alert
🤔 Your Challenge
If an alert fires this often and is consistently ignored, is it still doing its job — and how would you redesign it?
- What's the actual cost of an alert that fires 200 times a day and is always dismissed?
- If the condition usually resolves itself within minutes, does it need to page anyone at all?
- What would tying this alert's threshold to an SLO, instead of a raw metric value, change about how often it fires?
- Is there a real signal buried in these 200 firings that a smarter alert would actually catch?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
An alert nobody investigates isn't providing detection value anymore — it's providing habituation. The real cost of 200 daily firings that are routinely dismissed isn't just the interruption itself; it's what happens to attention over time. A responder trained by months of "this always resolves itself, dismiss it" will apply that same reflex the one time it doesn't self-resolve, and that's exactly the scenario a noisy alert quietly sets up.
Step 2 — Identify the likely bottleneck
A few common causes produce this pattern: a threshold set too sensitively, tripping on normal variance rather than a genuinely actionable condition; an alert on a raw metric value (a specific CPU percentage, a specific latency number) rather than on sustained SLO-relevant impact, which fires on every brief blip regardless of whether it actually threatens the SLO; or a condition that's genuinely self-healing by design (a transient retry, a brief autoscaling lag) that was never worth alerting on as an individual event in the first place, only as an aggregate trend if it happens too often.
Step 3 — Investigation
- Check what this alert's threshold is actually tied to — a fixed metric value, or something connected to SLO impact (error budget burn rate, sustained duration past a threshold)? This is usually the core design gap.
- Check the typical duration of the underlying condition before self-resolution — if it's consistently under a few minutes, that duration itself is informative: is there a reasonable case for not alerting on anything that resolves within it?
- Check whether any of the 200 daily firings have ever correlated with an actual incident or customer impact — if the answer is genuinely never, that's strong evidence the current threshold has no real signal value. If there's even an occasional real signal buried in the noise, that's worth identifying specifically rather than dismissing the whole alert.
- Check how this alert's firing rate compares to how often the team actually needs to intervene — a large gap between "fires" and "requires action" is the quantified version of the noise problem.
Step 4 — Recommended action
Redesign the alert around actual impact rather than a raw threshold: require the condition to persist for a meaningful duration (longer than its typical self-resolution time) before paging, so momentary, self-healing blips stop generating pages at all. Where possible, tie the threshold to SLO burn rate rather than an absolute metric value — this way the alert reflects "is this actually threatening our reliability commitment" rather than "did a number cross an arbitrary line," which tends to produce far fewer, far more meaningful firings.
For genuinely transient, self-healing conditions that are still worth tracking in aggregate (if they start happening unusually often, that itself might be meaningful), consider a dashboard or a trend-based alert on frequency instead of an individual page per occurrence.
Step 5 — Engineering lesson
An alert's value isn't measured by how reliably it detects a condition — it's measured by how reliably it gets appropriate action from the people it pages, and a condition dismissed 200 times a day, every day, isn't producing appropriate action, it's producing habituation. If the team's real, well-trained response to an alert is "ignore it," the alert has already failed at its actual job, regardless of how technically accurate its detection logic is.
🧠 Today’s Takeaway
If everyone ignores an alert, it isn't doing its job.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

