Can AI Create Production Alerts?

Step 1 — Understand the symptoms

The AI's output quality isn't really the question here — the scenario stipulates it's finding real gaps, and that's plausible; this is exactly the kind of pattern-matching-across-large-telemetry-volumes task LLMs tend to be genuinely useful for. The actual question is about the last step: recommending versus enabling.

A recommendation that's wrong costs someone a few minutes reviewing and rejecting it. An auto-enabled alert that's wrong costs someone a 3 a.m. page for nothing, or — worse — trains the team to start ignoring pages because too many of them turn out to be noise. Those are very different failure costs, and the proposal on the table skips straight past the cheap-failure version to the expensive one.

Step 2 — Identify the likely bottleneck

The risk isn't "the AI is bad at this." It's that alert quality depends on context an AI analyzing metrics, logs, and traces doesn't automatically have: what this specific service's acceptable error budget actually is, whether a given threshold is normal seasonal variance or a real problem, whether a metric spike correlates with a known, already-mitigated issue, and what the organizational cost of a false-positive page actually is for this specific team. A threshold that looks statistically reasonable from telemetry alone can still be operationally wrong — too sensitive for a team that already has alert fatigue, or too loose for something genuinely critical.

There's a second, quieter risk too: the alert-generation system itself can degrade — a change in the underlying model, a shift in what data it's being given, a subtle bug in how it computes thresholds — and if there's no human in the loop and no monitoring on the generation process itself, that degradation shows up first as production pages, not as a caught mistake.

Step 3 — Investigation

Before agreeing to any version of this, the questions worth asking are about the workflow, not the AI's competence:

• What does the approval path look like right now, and does "auto-enable" skip a step that currently exists for human-authored alerts too? If human engineers don't get to push straight to production alerting without review, AI shouldn't either — the standard should be at least as high, not lower, because there's no one person accountable for a specific threshold choice.
• Is there an SLO or error-budget context attached to each proposed alert, or is it purely statistical ("this metric looks unusual") without grounding in what actually matters for this service?
• What's the plan for noise? A batch of new alerts with untuned thresholds is a fast way to generate false positives at scale, right at the moment trust in the system is being established.
• Is the alert-generation process itself observable — versioned, auditable, with a way to see what changed if alert quality shifts over time?
• What's the rollback path if an auto-enabled alert turns out to be wrong — can it be disabled as fast as it was enabled, and is there a record of who (or what) enabled it and why?

Step 4 — Recommended action

Keep AI in the loop as a recommender, not an enabler, at least until there's real track record and real guardrails. A workable version of this: AI proposes alerts with reasoning attached (what pattern it saw, why it thinks this threshold), a human reviews and approves before anything goes live, new alerts start in a non-paging or shadow mode to validate signal quality against real production behavior before they can page anyone, and every AI-proposed-and-approved alert is versioned and auditable — so if something pages incorrectly, there's a clear record of what proposed it, who approved it, and what the reasoning was.

That's not "no automation." It's automation with a human accountable for the last step that actually matters — the one where a wrong decision means someone's sleep gets interrupted for nothing.

Step 5 — Engineering lesson

The useful question with AI-assisted SRE tooling usually isn't "is the AI good enough." It's "what's the blast radius of the AI being wrong, and does the workflow match that blast radius." AI generating a report is low-stakes if wrong. AI auto-enabling a production page is not — and the gap between those two is exactly where guardrails, human review, and auditability need to live, not somewhere the team gets to add later once the first bad page has already gone out.

The same discipline that applies to any production automation — least privilege, staged rollout, auditability, a fast rollback path — applies here too. AI doesn't get an exemption from that just because it's AI; if anything, a system that can act at machine speed and machine scale needs those guardrails more, not less.