Alert Design People Trust

The purpose of an alert is to interrupt a person. That is a strong claim on someone's attention, and it is only justified when the condition is real, actionable, and owned.

Most districts run 100–400 alerts a week, of which perhaps five required action. The rest teach everyone to ignore the pager, which means the one that mattered is ignored too.

Three tests for every alert

1. Is it real? Would a competent engineer agree the service is degraded?
2. Is it actionable? Is there something to do, written down?
3. Is it owned? Does a named person or team respond at 02:00?

If any answer is no, it is a dashboard panel or a log entry, not an alert.

Symptom, not cause

Cause-based (bad)	Symptom-based (good)
CPU > 80% on the gateway	HTTPS requests to the gateway fail or exceed 2 s
Interface errors > 0	Packet loss to a building exceeds 1% for 5 minutes
DHCP service stopped	Test client cannot obtain a lease
AP offline	Clients in that wing drop below −75 dBm for 10 minutes

Cause-based alerts fire when something is unusual; symptom-based alerts fire when something is broken. Unusual is common. Broken is not.

Thresholds that do not flap

Technique	Use
for: 5m	Require the condition to persist
Hysteresis	Clear at 2% loss, fire at 5%
Baseline-relative	Fire on 3σ from the same hour last week
Rate, not counter	Alert on derivative, never on the raw total
Grouping	One alert per site, not per port

Flapping is the fastest way to lose trust: a link that bounces every ten minutes produces forty notifications a day and no information.

Severity and routing

Severity	Meaning	Route
Page	Service down, or degrading for many users	Phone, immediately
Ticket	A user or small group is affected	Help desk queue
Dashboard	Informational trend	No notification

Three levels, three routes. Anything more becomes a taxonomy nobody remembers.

Write the runbook first

Each alert links to a page that answers:

• What does this mean?
• What is affected?
• What are the first three checks?
• What is the rollback or mitigation?
• Who else needs to know?

An alert without a runbook is a puzzle delivered at the worst possible time.

Worked example: 40 alerts to 8

A district's monitoring had 40 rules: CPU on every switch, memory on every server, ping to 200 devices, disk on file servers, and a set of vendor-specific traps.

Consolidation:

Removed	Why
Ping to 200 devices	ICMP rate limits produced false positives; synthetic checks replaced it
CPU on switches	Never actionable; kept as a dashboard panel
Per-port link alerts	Replaced by "building uplink down"
Disk > 80% on file servers	Kept, but with a 90-day trend alert instead

Added	Why
DNS resolution fails from two VLANs	The actual failure users report
DHCP test lease fails	Catches the 08:00 outage
Internet TTFB > 2 s for 5 min	User-visible latency
Certificate expiring in 14 days	Preventable outage
UPS on battery > 5 min	Building-level risk

Result: 8 alerts, all actionable, each with a runbook. The volume dropped by 95% and the signal went up.

What breaks in production

• Alerting on everything because it is easy to add a rule.
• No grouping, so one uplink failure produces 60 port alerts.
• No maintenance windows, so every planned change pages someone.
• Alerts without owners, which are nobody's job by definition.
• Severity inflation, where everything is "critical" and nothing is.
• No post-alert review, so the same false alarm fires for years.

Checklist

1. Every alert passes the real, actionable, owned test.
2. Alerts are symptom-based.
3. for: durations and hysteresis prevent flapping.
4. Grouping produces one notification per incident, not per object.
5. Each alert links to a runbook with first checks.
6. Maintenance windows suppress planned work automatically.
7. The alert list is reviewed quarterly, and anything unused is deleted.

Next

Alerting is what interrupts. Continue with Dashboards That Answer Questions, which is what people look at when they are already curious.