It’s 3:12 a.m. and the on-call engineer’s phone buzzes for the fourth time tonight. Same red badge, same “CRITICAL” label, same service name she’s seen all week. She reads half of it, recognizes the shape, and swipes it away. Twenty minutes later customers can’t check out.
The alert that mattered looked exactly like the forty that didn’t. That’s alert fatigue in one sentence, and most teams treat it as a monitoring problem: tune the thresholds, delete a few rules, move on. Thresholds matter. But a big share of the noise comes from product decisions. What an alert says, who gets it, how loud it is and what you can do from it are design choices, and most DevOps and security tools make them badly.
Alert fatigue is a design problem, not only a threshold problem
Google’s SRE team describes the failure mode well in Monitoring Distributed Systems: when pages come too often, people start to second-guess, skim, or ignore them, and eventually miss a real one hidden in the noise. Their rule is blunt. Every page should be actionable.
Read that as a product requirement and it changes the conversation. “Actionable” isn’t a property of the metric. It depends on whether the person receiving the alert understands what’s broken, believes it, and knows the next step. Two alerts can watch the same signal with the same threshold, and one gets acted on while the other gets muted forever. The difference is the interface around it.
Alerts vs. notifications: the line most products blur
PagerDuty’s alerting principles draw a clean distinction. An alert needs a human to do something. Everything else is a notification: useful context, but nothing anyone can act on right now.
Open almost any monitoring or security console and you’ll find both dumped into one feed, styled the same way, ranked by time. A certificate expiring in 60 days sits next to a database refusing connections. Users can’t tell them apart at a glance, so they stop looking at all of it. If your product does one thing about alert fatigue, make it this: give alerts and notifications separate homes, separate visual weight and separate delivery channels.
A four-question test for every alert
When we audit alerting in a B2B product, we run each alert type through four questions. It’s a fast way to see which ones deserve to interrupt a human.
1. Does someone need to act?
If nobody can change the outcome, it’s a notification. Log it, show it in a digest, keep it off the pager.
2. Does it need action now?
Urgency decides the channel. Wake-someone-up urgent goes to the phone. Fix-it-this-week goes to a ticket or a daily summary. Most products only have one volume setting: loud.
3. Who, specifically?
“The team” is not a recipient. Route by service ownership, and show the owner on the alert itself. An alert that reaches eight people often gets handled by none of them.
4. Can they act from the alert itself?
If the first step is opening three other tools to figure out what’s going on, the alert is a pointer, not an alert. The best ones carry enough context to decide, plus a direct link to the next move.
Alerts that fail question 1 or 2 get demoted. Alerts that fail 3 or 4 get redesigned.
Before and after: redesigning one noisy alert
Here’s a typical alert from an infrastructure dashboard:
Before: CRITICAL: High CPU on node-7 (94%)
It’s describing a cause, not a symptom. Is anyone affected? Is 94% unusual for this node? Is someone already on it? The engineer has to answer all of that before deciding whether to care, and at 3 a.m. the honest answer is usually “probably fine.”
After: Checkout API: p95 latency above 2s for 10 minutes, 3% of requests failing. Likely related: node-7 CPU saturation since deploy #4812 (18 min ago). Owner: Payments on-call. Actions: open runbook, roll back deploy, silence for 1h (reason required).
Same underlying data. The difference is order and framing: user impact first, probable cause second, owner and actions last. That’s the symptom-first approach the SRE book recommends, applied to the actual words and buttons on the screen.
Design patterns that cut alert noise
Group by incident, not by event
One failing dependency can trigger dozens of downstream alerts. Showing each one separately turns a single problem into a wall of red. Correlate related alerts into one incident, with the noisy children collapsed underneath it.
Make silencing explicit and accountable
People will mute noisy alerts no matter what you do. So design the mute. Ask for a reason, set an expiry by default, and show who silenced what. A silenced alert with a note (“known issue, fix ships Thursday”) is useful history. A silenced alert with no trace is a future outage.
Show each alert’s track record
This is the pattern I’d push hardest for. Next to every alert rule, show how often it fired in the last 30 days and how often anyone took action. An alert that fired 46 times and led to action twice is telling you something, and putting that number in front of the people who own the rule does more than any cleanup sprint.
Let severity mean something
If everything is critical, nothing is. Keep severity levels few (three is usually enough), define each one in plain language tied to user impact, and reserve red for the top level only. Then hold the line. Severity creep is a design debt like any other.
Security products have it worse
In security tooling the volume is higher and the cost of a miss is bigger. SOC analysts triage endless streams of detections, many of them false positives, and the same fatigue sets in. Everything above applies, with one addition: explain why something was flagged. An analyst who can see the rule, the evidence and the confidence in one view decides faster and trusts the tool more. We wrote more about this in human-centered design in cybersecurity products.
How to know if your alert redesign worked
Don’t measure it by how calm the dashboard looks. Track behavior:
- Action rate: the share of alerts that led to a real action. This is the closest thing you have to a signal-to-noise ratio.
- Pages per on-call shift: trending down without missed incidents is the goal.
- Time to acknowledge: trusted alerts get answered faster.
- Silence volume: lots of long or reason-less silences point straight at the alerts that need work.
If you’re seeing broader friction in how your platform handles incidents, our breakdown of the UX pitfalls of DevOps platforms covers the surrounding workflow.
Where to start
Pick your ten loudest alerts and run them through the four questions. You’ll probably demote half of them and rewrite the rest. That alone buys back a lot of trust from the people carrying the pager.
The harder part is fixing it at the product level, so new alerts are born well-designed instead of cleaned up later. That takes someone who understands both the infrastructure and the interface. If your team is deciding whether to build that capability in-house or bring in a partner who has done it in Cloud, DevOps and security products before, let’s talk.
