Alerting replacements fail in a way that is almost invisible. You stand up the new stack, migrate your rules, turn off the old system, and everything looks quiet. Nobody gets paged for two weeks. You declare success.
Then something breaks that your old system would have caught, and you find out that the new system was never actually watching that thing. The silence wasn’t health — it was a gap you hadn’t proven didn’t exist.
The way to avoid this is to treat an alerting migration the same way you’d treat any production service migration: run both in parallel, prove parity, then cut over.
The instinct that causes the problem
When alerting feels painful — too much noise, wrong thresholds, rules that haven’t been touched since 2019 — the instinct is to build something better from scratch and replace the old thing in one move. That’s understandable. Carrying broken alerting while you build the replacement is unpleasant.
But “replace everything at once” has a property you don’t want: it makes it impossible to know what you’ve lost until you lose it.
Every alerting system has coverage gaps. The new one will too — different gaps, maybe smaller, but gaps nonetheless. The only way to find them before an incident does is to run the old and new systems in parallel long enough that real production conditions exercise them both.
Shadow before you shift
The pattern is the same as dark traffic testing for services. For some period — weeks, not days — send all alert signals to both systems. The old system pages as normal. The new system fires its alerts into a log or a separate low-priority channel where you can see them without being woken up.
Then you compare. For every page the old system fired: did the new system also fire? Did it fire with better signal or worse? Did it fire sooner or later?
And crucially: for things the new system is firing on that the old one isn’t, you review those too. Some will be legitimate gaps in the old coverage. Some will be false positives in your new rules that need tuning before the new system is in primary position.
On a platform migration I worked on — replacing a legacy commercial monitoring stack with Prometheus, Grafana, and Wazuh — we ran the shadow period for about four weeks before shifting a single production component to the new stack as primary. In those four weeks we found three categories of gaps: things the old system fired on that we hadn’t ported, things we’d ported with wrong thresholds that would have produced noise, and one critical path — storage latency on persistent volumes — that neither system was covering well and that we would have missed entirely if we’d just done a straight cutover.
Retire rules incrementally, not all at once
Once you have confidence that the new system covers a given component, turn off the old rules for that component — not the whole old system. Work component by component, or team by team.
This matters because it preserves your ability to compare. If you retire all the old rules at once, you’ve lost your baseline. If you retire them incrementally, you can continue comparing across the boundary for the components that haven’t switched yet.
It also surfaces the rules nobody wanted to own up to. When you go through a legacy alerting system rule by rule, you find alerts that nobody can explain, nobody depends on, and nobody wants to move. Those shouldn’t be migrated. They should be deleted. The incremental process forces that conversation.
Silence is not a test
The thing to resist is treating a quiet week as confirmation. A quiet week during parallel operation might mean everything is working. It might mean your new system isn’t seeing the events it needs to see. It might mean production was unusually stable and nothing that would trigger an alert actually happened.
The validation you want is coverage proof: for a given component, over a given time window, the new system fired on everything the old system fired on — and you can show it. Not “nothing paged,” but “these specific events were observed by the new system and here is how it classified them.”
That kind of proof is harder to produce than “it’s been quiet.” It requires keeping records, doing comparisons, and being willing to say “we haven’t proven coverage on this component yet, so we’re not cutting it over.” But it’s the only proof that means anything.
The broader point
The discipline here isn’t about observability specifically. It’s about not treating a replacement as proof of its own correctness. Any system you put in place to watch something else is only as trustworthy as your evidence that it’s watching. Running both systems in parallel, comparing outputs, and retiring the old one incrementally is how you generate that evidence.
The teams that do this well treat observability migrations with the same rigor they’d apply to a database migration or a load balancer swap. The teams that don’t find out the hard way that quiet isn’t the same as covered.