I’ve never had an unplanned production outage on a migration I led. Ten years, four clouds, one bare-metal Kubernetes platform, one ad platform validated at 4.5M requests per second.
I’d love to say that’s skill. Mostly it’s a habit: I don’t trust a safety mechanism I haven’t watched work.
Documented is different from rehearsed
Every runbook I’ve inherited had a rollback section. Most of them said something like “redeploy previous version”. No one had run it in months. Sometimes the previous version’s image had been garbage-collected.
That’s a wish with a heading.
A rehearsed rollback has been executed, on infrastructure that looks like production, recently enough that the person on shift has seen it succeed. At a securities brokerage in Indonesia, where I ran phased cutovers of about 100 applications onto a self-managed Kubernetes platform, every traffic step on the way to 100% had a rollback we’d already run at the previous step. The cutovers were dull. Dull was the goal.
Controls beat heroics
The other half is designing so that the safe action is also the lazy action. On that same platform we put in place:
- Dual-approval deploys to production. One person can’t ship alone at 2am.
- Environment protection rules so a pipeline for staging physically can’t target prod.
- Deploy freezes for known-risky windows, enforced in the pipeline, not in a calendar invite.
- Drift detection, so a console change that bypassed Terraform shows up as a diff, not as a surprise six weeks later.
- Audit logging on everything above, so “what changed” is a query.
None of this is clever. Each control removes one class of 2am decision. That’s the point: reliability engineering is largely the removal of decisions from the moments when people are worst at making them.
Page a human only when a human can act
Alerting on that platform was tuned with one rule: if the page fires, the person woken up must have something to do. Disk at 80% on a node the autoscaler will replace? Dashboard, not page. Certificate expiring in 6 days? Ticket, not page. Error rate on the trading-adjacent API above threshold for 3 minutes? Page.
I’ve watched teams with beautiful SLO dashboards burn out because every alert was a page and every page was noise. The dashboard didn’t cause the burnout, but it didn’t prevent it either. Alert hygiene did.
The honest edge
I’ve built the guardrails that make on-call humane: the approvals, the freezes, the drift checks, the actionable-only paging. I haven’t personally carried the pager for a service I owned as an SRE, and I don’t pretend otherwise in interviews. It’s the specific thing I want to add next.
But I’d argue the sequence matters. Put someone on-call for a platform without rehearsed rollbacks and enforced controls and you’ve built a very expensive alarm clock. Do the upstream work first. Then the pager becomes bearable, and SLOs become something you can actually defend.
If you can’t rehearse it, you don’t have it. You have a document.