An alert is binary by design. It resolves one question, which is whether something is wrong enough to interrupt a person. A dashboard does something else. It shows what the system is doing across a window of time, and lets the engineer decide whether what they are seeing matches what they expect. Alert tuning, no matter how careful, will never produce that. It produces a more reliable threshold, which is a different artifact entirely.

The first job of a dashboard is to shorten the distance between the page and the diagnosis. When an alert fires, an engineer opens a dashboard before doing anything else. The alert supplies the interrupt. The dashboard supplies the context that turns the interrupt into a working hypothesis. A team with disciplined alerts and neglected dashboards produces an on-call experience where engineers know they need to act but spend the first ten minutes assembling the picture by hand. That time compounds across every incident the team handles.

The picture they assemble by hand is not free. An engineer without a dashboard reaches for logs or traces, which means querying a log store, scrolling through trace spans, or in the worst case opening a shell on a worker to read process state directly. Each of those is more expensive than reading a pre-computed metric. Logs and traces are high-cardinality records of individual events, retained at a cost that scales with traffic. Metrics aggregated into a dashboard are cheap to store and cheap to query, because the aggregation has already happened. A team that operates without dashboards is implicitly choosing to do that aggregation by hand, in real time, against the most expensive data store they have, while paying to retain that data long enough to be useful. A well-designed metrics layer lets a team reduce log and trace retention to the window where those tools actually earn their cost, which is investigating the recent past in detail. The dashboard carries the long view. Logs and traces carry the close view. Conflating the two pushes cost into the storage tier that is least suited to bear it.

The second job of a dashboard is to surface conditions that never reach an alert threshold. A well-tuned alert fires when something is unambiguously wrong. Many real failure modes develop gradually. Latency creeps. Error rates rise by tenths of a percent per week. Queue depth climbs over a quarter until a Tuesday afternoon spike pushes it over. None of these cross an alert threshold on the day they begin, and a system that relies only on alerts will not see them until they do. A dashboard that someone reads habitually, the way a pilot reads instruments, catches the trend in the weeks before the threshold. This is work that alerts cannot do, because alerts exist to suppress information until a specific condition is met.

The third job of a dashboard is to encode the team’s mental model of the service. Choosing what to display, in what arrangement, at what time scale, is an act of describing how the team believes the system behaves. A dashboard with stale panels, metrics nobody reads, or queries that no longer return data reflects a team whose shared model has decayed. The dashboard becomes a record of what the service used to be rather than a window onto what it is. This is harder to detect than a noisy alert because nothing fires when a dashboard goes stale. It simply stops being consulted, and the team adapts to operating without it.

Good dashboards therefore require their own form of gardening. The work is not identical to alert maintenance but the discipline is the same. Panels need to be removed when the metric they show no longer corresponds to a question the team asks. New panels need to be added when the team starts caring about a new dimension of the service. Time ranges need to be revisited as the rhythm of the service changes. The cost of skipping this maintenance is not a false page. It is a slow drift toward a dashboard that engineers open out of habit and close without learning anything.

Consider a worked example. An alert fires for elevated error rates on a checkout service. The engineer on call opens the service dashboard and sees that errors are concentrated on requests routed through one of three upstream payment providers, that latency to that provider has roughly doubled over the past forty minutes, and that the error rate on the other two providers is flat. The diagnosis at that point is essentially complete. Without that dashboard, the same engineer would be querying logs for error patterns, comparing timestamps, and trying to reconstruct the upstream distribution from individual log lines. The alert would have fired in either case. The difference between a five minute resolution and a forty minute resolution is whether the dashboard already existed when the page came in.

A defender of the alerts-only position might respond that the fix is one alert per provider, so that the comparison is implicit in which alerts have fired. That works at three providers and breaks at thirty. It also pushes the comparison into the alert layer, which is the layer least suited to carry it, and it adds thresholds to maintain where one dashboard panel would do.

Capacity planning is the other side of the same coin and rarely involves a page at all. The signal that matters is rarely the current state. It is the trajectory of the system over months, read against the trajectory of the business. A team that has been recording disk utilization, compute pressure, and queue depth as metrics for the last year can answer questions of the form: my disk has filled at five percent per month linearly, my average compute utilization has moved from seventy-five percent to eighty percent over the last six months, and we have onboarded a thousand customers at an average size of x during that period. Those three sentences together produce a defensible forecast for the next region launch or the next sales push. Without the metrics layer, that data may simply not exist. Logs and traces age out, often within weeks, because retaining them at the volumes a busy service produces is expensive. Pulling a one-off chart from log data assumes the log data is still around to pull from, and at six or twelve month horizons it usually is not. The team is then forced to forecast from what people remember about how the system behaved, which is a worse input than a chart and a more confident one than it deserves to be.

A team with strong alerts and weak dashboards knows when to act and spends the first ten minutes of every incident reconstructing context. The inverse team, with rich dashboards and thin alerting, often hears about its outages from customers first. Neither failure mode is rare, and both are predictable consequences of treating one practice as a substitute for the other. Alerts and dashboards belong to the same garden. They do not grow the same plants.

In a well-resourced environment, alerts derive from customer signals such as throughput, latency, and rate-limit consumption rather than from hardware proxies like disk space or CPU utilization. Product owners define the thresholds at which latency or error rates constitute a customer-visible problem. Performance testing establishes the IO wait percentage or kernel object counts a service can tolerate before degradation. These conditions describe a mature alerting practice, and most teams do not begin there. Engineers inherit legacy services, absorb new traffic patterns from upstream callers, and ship features faster than benchmarks can catch up. The thresholds set on day one rarely match the system six months later, and the gap between intent and reality widens with every deploy.

This drift is the core problem. An alert calibrated for last quarter’s traffic will either fire constantly or stay silent through real incidents. A threshold chosen without performance data reflects a guess, and guesses age poorly. New user flows shift the baseline. Dependencies change their failure modes. The signals that mattered at launch may no longer correlate with customer pain, and the only remedy is the steady work of revisiting each alert against current conditions, removing the ones that no longer indicate harm, and reordering the rest by the severity of impact they actually represent.

The principle that should guide this work is intolerance of false positives. A page at three in the morning for a non-issue costs more than the lost sleep. It erodes the engineer’s productivity for the following day, damages morale across the rotation, and, most consequentially, trains the responder to discount the next page. An engineer who has been burned by a false alarm at three is slower to react to a real incident at six, more likely to silence the notification, and in the worst case will sleep through the page entirely. False positives therefore deserve the same treatment as outages, which is prompt investigation, labeling, and adjustment.

The discipline this requires cuts both ways. Widening a threshold is the easiest response to a noisy alert and the one most likely to produce a silent miss. A threshold tuned too loosely means the first signal of a real incident arrives as a customer complaint, which is the outcome alerting exists to prevent. The correct response to a false positive is rarely to relax the threshold by a fixed margin, but to ask why the alert fired without a real problem behind it. The answer might point to a missing condition, a noisy upstream dependency, a metric that needs smoothing, or an alert that should be replaced by something closer to the customer experience. Each of these is a more durable fix than a wider tolerance, and each leaves the rotation better calibrated than it was.

Tuning a specific alert involves two dimensions that teams often conflate. The first is the threshold itself, the value at which the metric crosses from acceptable to actionable. The second is the duration the metric must hold that value before the alert fires. A latency alarm that pages on a single elevated data point will catch transient spikes that resolve before the engineer opens a laptop. The same alarm configured to fire only after five minutes of sustained elevation will ignore the spike but catch the genuine degradation. Adjusting duration is often the higher-leverage fix because it filters noise without sacrificing sensitivity to the underlying condition. Threshold adjustment, by contrast, changes what the alert considers a problem at all, and should follow from evidence about the service’s real behavior rather than from a desire to quiet the rotation. A team that tunes both dimensions deliberately will produce alerts that fire when something is genuinely wrong and stay silent when the system is merely noisy.

There is a further category of fix that teams often overlook, which is to change the application rather than the alert. When a service produces conditions that the team has decided to treat as false positives in perpetuity, the alert is reporting something real about the application’s behavior, even if the team has chosen not to act on it. A nightly batch job that drives queue depth past the alarm threshold every night, a retry loop that briefly saturates a connection pool on every deploy, or a memory pattern that triggers a warning every time garbage collection runs are not alerting problems. They are application problems that the team has agreed to ignore. The honest response is to fix the underlying behavior, because an application that requires its operators to maintain a standing list of conditions to disregard is one in which real degradation will eventually hide among the conditions everyone has learned to wave away.

Alerting is therefore not a configuration task but a continuous conversation between a team and its service. The thresholds, durations, and signals that constitute a healthy rotation reflect what the team currently knows about how the service fails and how those failures reach customers. That knowledge changes, and the alerts must change with it. Teams that schedule regular alert review, treat false positives as defects to investigate rather than nuisances to suppress, and remain willing to fix the application when the application is what is broken will build rotations that protect both their customers and the engineers who carry the pager. Teams that do not will discover, usually at three in the morning, that the garden has gone to seed.