Every observability vendor now has a demo where a language model reads your alerts, correlates the logs, and drafts the incident summary. Some of them are good. All of them share a property that never makes the slide: the model is downstream of your alert quality.
Feed it a page for every disk at 80% and every flapping health check, and you’ve built a very fluent way of being woken up for nothing.
Alert hygiene comes first, and it’s a design job
On a self-managed Kubernetes platform for a securities brokerage in Indonesia, we set observability up with one rule for paging: a human is woken only when a human has something to do. Everything else is a dashboard, a ticket, or an automated action. Prometheus and Grafana for the metrics side, Wazuh for security events, and a deliberate, argued-over list of what actually pages.
That took real effort. Every “just add an alert” request got the question: what would the responder do at 3am? If the answer was “look at it and go back to sleep”, it didn’t page.
An LLM layered on that alert stream has a fighting chance. Layered on the average alert stream, it’s summarising noise.
Automate the deterministic responses before you add a model
My first auto-remediation was in 2018, on a HIPAA-compliant healthcare platform on AWS: Python on Lambda that replaced failed nodes and handled the outage-response steps we’d already written down in runbooks. Nothing intelligent about it. It was the runbook, executed by a function instead of a tired person.
That’s still the right order. If a response is deterministic (restart this, replace that, fail over here, roll back to the last good release under conditions we’ve defined), it should be code, tested, with an audit log. Handing it to a model adds latency and a new failure mode to something that didn’t need either.
What’s left after that pass is the ambiguous stuff. That’s where a model helps: pulling the last three deploys, the related alerts, and the relevant runbook into one place, and drafting the first message to the channel while the responder is still logging in.
Where I’d let the model in
- Triage summaries: what changed, what’s correlated, what’s the likely blast radius. Drafted, then read by a human.
- Runbook retrieval: the right procedure, surfaced at the right moment, from a knowledge base the team maintains. (I’ve done the knowledge-base half with NotebookLM for distributed delivery teams; the pattern transfers.)
- Postmortem first drafts, from the timeline the tooling already recorded.
Where I wouldn’t: taking any production action without a human approving it. Same rule I apply to agents writing infrastructure code. Author, don’t approve.
The controls the model inherits
The reason I’m relaxed about AI in incident response on platforms I’ve built is that the platform already had dual-approval deploys, deploy freezes, drift detection, audit logging, and rehearsed rollbacks before any model showed up. The model can propose. The controls decide what’s possible.
If your platform doesn’t have those yet, an LLM in the loop doesn’t add safety. It adds a confident narrator to an incident you were always going to have.
Fix the pager. Automate the boring. Then invite the model, and keep the approval.