The "three pillars" model — metrics, logs, traces — comes from application observability. Networks have metrics and logs in abundance, and almost no traces, because a packet does not carry a request ID. What transfers is the discipline of choosing which one answers which question.
What each is good at
| Signal | Strength | Weakness | Network example |
|---|---|---|---|
| Metric | Cheap, aggregatable, long retention | No context | Interface utilization |
| Log | Context, discrete events | Volume, parsing | Auth failure, link flap |
| Trace | Causality across components | Requires instrumentation | Synthetic transaction |
A trace answers "where did the time go in this request." In a network, the closest equivalent is a synthetic transaction that walks DNS → TCP → TLS → HTTP and times each step. That is worth building, because it produces the same causal story without instrumenting the application.
Cardinality is the hidden cost
A metric is cheap when it has few label combinations and expensive when it has many. ifOperStatus
per interface on 400 switches is fine. bytes_per_source_ip on 3,000 devices is a cardinality
explosion that will take down a time-series database.
Rules of thumb:
- Label by interface, site, and role — not by client IP.
- Keep high-cardinality data in logs or flows, where it is indexed rather than multiplied.
- Aggregate before storing when you do not need per-object detail.
Averages versus percentiles
This is the single habit that fixes most dashboards:
| Statistic | What it tells you | What it hides |
|---|---|---|
| Mean | The typical value | The twenty bad minutes |
| p50 | The median experience | The tail |
| p95 | What the slowest 5% endure | Nothing important |
| p99 | The worst 1% | Nothing important |
| Max | The single worst moment | Whether it was one packet or one hour |
For latency and utilization, p95 is the working number. For capacity planning, the trend of p95 over 90 days is better than any single peak.
A mean latency of 20 ms can contain a 4-second tail
If 1% of requests take four seconds, the mean barely moves and 30 users a day have a terrible experience. Percentiles are how you see them.
Sampling and aggregation
- Counters should be stored as rates, computed from deltas, with wrap handling.
- Latency should be stored as a histogram if you want percentiles later; a pre-computed mean cannot be un-averaged.
- Logs should be sampled only when volume forces it, and the sampling rate recorded.
Worked example: the dashboard that lied
A dashboard showed average WAN latency of 18 ms and a healthy green status. Users complained every morning.
Recomputed as p95, the same data showed 900 ms between 07:40 and 08:10. The mean was dragged down by 19 hours of idle time. One percentile change turned a "no problem" dashboard into an accurate one.
What breaks in production
- Dashboard by average, hiding the peak that users experience.
- High-cardinality metrics, filling the database and slowing queries.
- Traces assumed to exist in a network where nothing is instrumented.
- Logs used as metrics (counting lines to compute a rate), which is expensive and fragile.
- No synthetic transaction, so "the network is up" never becomes "the service works."
Checklist
- Each dashboard states which signal it uses and why.
- Latency and utilization use percentiles.
- Cardinality is bounded by design, not by accident.
- A synthetic transaction measures the user path end to end.
- Logs are used for events, not for rates.
- Retention differs by signal, deliberately.
Next
You can measure the right things. Continue with Alert Design People Trust, where most monitoring systems fail.