Alert Fatigue: Symptom-Based Alerting That Works

Your on-call engineer gets paged at 2 AM for high CPU on a batch processing node. They investigate for 20 minutes, find nothing wrong, and go back to sleep. An hour later, another page: disk space on a log aggregator. Another false alarm. By the time the third alert fires — this one signaling elevated API error rates — they’ve learned to assume it’s another false positive. They check Slack to see if anyone else is handling it, see nothing, and go back to sleep. In the morning, they discover the payment service was down for 45 minutes. Customers couldn’t complete purchases. The alert was real; the engineer had just been trained to ignore it.

This is alert fatigue in action. When everything alerts, nothing alerts. Teams learn to ignore pagers that cry wolf, and real incidents hide in the noise.

The Alert Fatigue Problem

I’ve seen the same pattern at every company with a mature monitoring stack: start with a few carefully designed alerts, watch them multiply through incident post-mortems (“we should add an alert for that”), and end up with hundreds of alerts that nobody trusts. The progression is predictable.

At first, you have two or three alerts per day, and 90% require real action. Engineers investigate every one thoroughly. Then alert creep sets in — seven or eight alerts per day, 50% actionable. Response time increases as people start triaging instead of investigating. By the time you hit 30+ alerts per day with only 20% requiring action, teams batch-acknowledge and selectively respond. Real incidents get missed in the noise. Eventually, engineers mute channels and ignore pages entirely. The alerting system is effectively abandoned.

Breaking this cycle requires three changes. First, alert on symptoms that affect users rather than causes that might affect them — this dramatically reduces false positives. Second, require runbooks for every alert so responders can act quickly and confidently. Third, conduct regular alert hygiene reviews to prune noise and retire alerts that no longer earn their keep. These aren’t optional practices for mature teams; they’re prerequisites for an alerting system that actually works.

Measuring Alert Quality

You can’t fix what you don’t measure. Before optimizing your alerts, establish baseline metrics that reveal whether your alerting helps or hurts.

The actionable rate is the most important metric. If fewer than 80% of your alerts require human intervention to resolve, you’re training your team to ignore the pager. Track this weekly and treat declining actionable rates as a P2 issue.

Symptom vs Cause Alerting

The most common alerting mistake is alerting on causes rather than symptoms. High CPU, low disk space, elevated connection counts — these are causes. They might affect users, or they might not. Request latency, error rates, failed transactions — these are symptoms. They tell you users are actually experiencing problems right now.

The Symptom-Based Philosophy

Consider three classic cause-based alerts and their symptom-based alternatives:

The pattern: causes are potential problems; symptoms are actual problems. Page on symptoms, ticket on causes.

Deriving Alerts from SLOs

If you have SLOs, you already know what symptoms matter — they’re the SLIs that feed your objectives. SLO-based alerting flips the traditional model: instead of asking “is this metric above a threshold?”, you ask “are we burning through our error budget faster than we can afford?”

Consider a 99.9% availability SLO. That gives you a 0.1% error budget per month — roughly 43 minutes of downtime. The question isn’t whether errors are happening (they always are), but whether they’re happening at a rate that threatens your SLO.

Burn Rate Fundamentals

Burn rate measures how fast you’re consuming your error budget relative to a sustainable pace. A 1x burn rate means you’ll exactly exhaust your budget by month’s end — sustainable but leaves no margin. A 14.4x burn rate means you’ll exhaust your entire monthly budget in just 2 hours. That’s an emergency.

The math: if your monthly budget is 0.1% errors and you’re currently seeing 1.44% errors (14.4 × 0.1%), you’re burning 14.4x faster than sustainable. At that rate, your 30-day budget disappears in 30 days ÷ 14.4 = ~2 hours.

This framing changes how you think about alerts. A 1% error rate might sound scary in isolation, but if your SLO allows 0.1% errors, that’s a 10x burn rate — you have 3 days before budget exhaustion. Urgent, but not a 2 AM page. A 0.3% error rate is only 3x burn, giving you 10 days. That’s a ticket for business hours.

groups:
  - name: slo-burn-rate-alerts
    rules:
      # Fast burn: 14.4x rate exhausts budget in 2 hours
      - alert: APIAvailabilityFastBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API error rate burning SLO budget rapidly"
          runbook_url: "https://runbooks.example.com/api-availability"

      # Slow burn: 3x rate exhausts budget in 10 days
      - alert: APIAvailabilitySlowBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            / sum(rate(http_requests_total[1h]))
          ) > (3 * 0.001)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "API error rate elevated"
          runbook_url: "https://runbooks.example.com/api-availability"

Multi-Window Alerting

Single-window alerts have a problem: short windows catch spikes but also false-positive on brief blips; long windows miss fast-moving incidents. The solution is requiring both a short and long window to breach before alerting.

For a 14.4x burn rate alert, require both the 5-minute and 1-hour windows to show excessive errors. This prevents two failure modes: a 1-minute spike won’t trigger if the 1-hour window is healthy, and a long-resolved issue won’t keep alerting if the short window has recovered.

Google’s SRE book calls this “multi-window multi-burn-rate alerting,” and it’s the gold standard for SLO-based alerts. Note that 1x burn rate doesn’t appear here — if you’re only burning at the sustainable rate, that’s a dashboard metric, not an alert.

Alert Signal Selection

Not all metrics make good alert candidates. The four golden signals — latency, traffic, errors, and saturation — provide a framework for choosing what to alert on and, equally important, what not to alert on.

Golden Signals as Alert Candidates

Absence of Data Alerts

One category of alert often forgotten: detecting when monitoring itself fails. If your metrics stop arriving, your symptom alerts won’t fire — not because the service is healthy, but because you’ve gone blind.

Absence alerts detect gaps in your telemetry. The challenge is distinguishing “no data because nothing is happening” from “no data because collection is broken.”

groups:
  - name: monitoring-health
    rules:
      # No requests during US business hours = likely outage or collection failure
      - alert: NoRequestsReceived
        expr: |
          absent(rate(http_requests_total[5m]))
          and on() (hour() >= 14 and hour() <= 22)
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "No requests received - possible collection failure or outage"

      # Metrics haven't updated in 2 minutes = stale scrape
      - alert: StaleMetrics
        expr: |
          time() - prometheus_target_scrape_last_success_timestamp_seconds > 120
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Metrics stale for {{ $labels.instance }}"

      # Expected target missing (unless intentionally scaled to zero)
      - alert: TargetMissing
        expr: |
          absent(up{job="api-server"})
          unless on() kube_deployment_spec_replicas{deployment="api-server"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API server target missing from Prometheus"

Threshold Tuning

The hardest part of alerting isn’t deciding what to alert on — it’s deciding when. Set thresholds too low and you get noise; set them too high and you miss incidents. The goal is finding the sweet spot where most alerts correlate with real problems.

Data-Driven Threshold Selection

Don’t guess at thresholds. Analyze your historical data to find where problems actually occur.

Start by looking at the distribution of your metric over the last 30 days. What’s the P50 (typical)? P95 (elevated)? P99 (unusual)? Then correlate with past incidents: what were the metric values during actual outages? A good threshold sits between “normal high” (P99 of typical operation) and “incident low” (the minimum value during past incidents).

Three indicators tell you if your threshold is calibrated correctly:

Review thresholds quarterly. As your system scales, yesterday’s anomaly becomes today’s normal.

Dynamic Thresholds

Static thresholds break when baselines change. Traffic doubles during a product launch; latency improves after an optimization; seasonal patterns shift with business cycles. Dynamic thresholds adapt automatically.

The following examples assume you have recording rules pre-computing P99 latency (e.g., http_request_duration_seconds:p99:rate5m). If not, substitute the full histogram_quantile expression.

groups:
  - name: dynamic-alerts
    rules:
      # Alert when latency exceeds 3 standard deviations from 1-hour average
      - alert: LatencyAnomaly
        expr: |
          (
            http_request_duration_seconds:p99:rate5m
            - avg_over_time(http_request_duration_seconds:p99:rate5m[1h])
          )
          / stddev_over_time(http_request_duration_seconds:p99:rate5m[1h]) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Latency anomaly detected (3+ stddev from baseline)"

      # Alert when traffic drops 50% compared to same time last week
      - alert: TrafficAnomalyVsLastWeek
        expr: |
          (
            sum(rate(http_requests_total[5m]))
            - sum(rate(http_requests_total[5m] offset 7d))
          ) / sum(rate(http_requests_total[5m] offset 7d)) < -0.5
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Traffic down 50%+ compared to last week"

Each approach has tradeoffs:

Runbook Integration

An alert without a runbook is a puzzle without instructions. At 2 AM, your on-call engineer shouldn’t have to reverse-engineer what the alert means, where to look, or how to fix it. Every alert needs a runbook that answers four questions: What’s the user impact? How do I diagnose it? How do I fix it? When do I escalate?

Runbook Structure

A good runbook follows a consistent structure that an engineer can navigate under pressure.

Here’s a complete example:

# APIHighErrorRate

## Summary

- **User Impact**: Users experiencing 5xx errors on API requests
- **Time to Resolve**: 15-30 minutes typically
- **Escalation**: Page backend-oncall if not resolved in 30 min

## Diagnosis

### Key Questions

1. Is this affecting all endpoints or specific ones?
2. Did a deployment happen recently?
3. Are downstream dependencies healthy?

### Dashboard Links

- [API Error Rate Dashboard](https://grafana/d/api-errors)
- [Dependency Health](https://grafana/d/dependencies)
- [Recent Deployments](https://deploy.internal/history)

### Log Query

```text
service:api-gateway level:error | stats count by endpoint
```

### Common Causes

| Cause | Likelihood | Check |
|-------|------------|-------|
| Bad deployment | High | Check deploy history, rollback |
| Database overload | Medium | Check DB dashboard |
| Downstream timeout | Medium | Check dependency health |
| Memory pressure | Low | Check pod memory usage |

## Remediation

### Step 1: Identify Scope

```bash
kubectl logs -l app=api-gateway --tail=100 | grep -i error
```

**Expected**: Error pattern becomes clear

### Step 2: If Recent Deployment

```bash
kubectl rollout undo deployment/api-gateway
```

**Expected**: Error rate returns to normal within 2 minutes

### Step 3: If Database Issue

- Scale read replicas: `kubectl scale deployment/db-replica --replicas=5`
- Enable circuit breaker: Set `DB_CIRCUIT_BREAKER=true` in config

### Verification

- Error rate below 0.1% on [dashboard](https://grafana/d/api-errors)
- No new error logs in last 5 minutes
- User reports in #support channel resolved

## Escalation

- Page `backend-oncall` if: Not resolved in 30 min, error rate > 10%, affecting payments
- Page `database-oncall` if: Database-related and DBA intervention needed
- Page `incident-commander` if: Customer-facing outage > 1 hour

## Post-Incident

- [ ] Create incident ticket
- [ ] Update this runbook if new cause discovered
- [ ] Schedule post-mortem if P1/P2

Alert-Runbook Linking

Every alert must include a runbook_url annotation that links directly to its runbook. This creates a contractual relationship: if you add an alert, you must also write a runbook.

groups:
  - name: api-alerts
    rules:
      - alert: APIHighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "API error rate is {{ $value | humanizePercentage }}"
          runbook_url: "https://runbooks.example.com/api-high-error-rate"
          dashboard_url: "https://grafana.example.com/d/api-errors?var-window=5m"

Enforce this in CI. Before an alert rule can be merged:

Stale runbooks (older than 90 days) should automatically generate review tickets. A runbook that references deprecated tools or outdated procedures is actively dangerous — it sends responders down wrong paths at the worst possible time.

Alert Routing and Escalation

Getting the right alert to the right person at the right time is as important as the alert itself. A critical page at 3 AM should wake someone up; a warning about slow degradation should create a ticket for business hours. Misrouted alerts train people to ignore the pager.

Routing Configuration

Alertmanager (or your alerting platform of choice) routes alerts based on labels. The most common routing dimensions are severity, owning team, and time of day.

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts: page immediately, repeat every 15 minutes
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 0s
      repeat_interval: 15m

    # Warning alerts during business hours: Slack
    - match:
        severity: warning
      receiver: 'slack-warnings'
      active_time_intervals:
        - business-hours

    # Warning alerts outside business hours: ticket only
    - match:
        severity: warning
      receiver: 'ticket-system'
      active_time_intervals:
        - outside-business-hours

    # Team-specific routing: payments team has its own escalation
    - match:
        team: payments
      receiver: 'payments-team'
      routes:
        - match:
            severity: critical
          receiver: 'payments-pagerduty'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<pagerduty-key>'
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
        details:
          runbook: '{{ .CommonAnnotations.runbook_url }}'
          dashboard: '{{ .CommonAnnotations.dashboard_url }}'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: '<slack-webhook>'
        channel: '#alerts-warnings'
        title: '{{ .CommonAnnotations.summary }}'
        text: |
          *Runbook*: {{ .CommonAnnotations.runbook_url }}
          *Dashboard*: {{ .CommonAnnotations.dashboard_url }}

time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'
        location: 'America/New_York'

  - name: outside-business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '00:00'
            end_time: '09:00'
          - start_time: '18:00'
            end_time: '24:00'
      - weekdays: ['saturday', 'sunday']

Key routing principles:

Escalation Policies

Escalation ensures that unacknowledged alerts reach someone who can respond. The policy defines who gets paged first, how long to wait, and who to escalate to if there’s no acknowledgment.

A typical escalation chain for critical production issues:

Escalation policies should also handle auto-resolution. If the metric recovers and stays healthy for a reasonable period (typically 1 hour), the incident can auto-close without requiring explicit resolution. This prevents orphaned incidents that technically resolved themselves but remain open in the system.

Alert Hygiene and Maintenance

Alerts decay. Thresholds that made sense six months ago may be too sensitive or too lax today. Runbooks reference deprecated tools. Services get renamed. Without active maintenance, your alerting system becomes a source of noise rather than signal.

Regular Alert Reviews

Schedule monthly alert reviews. The goal isn’t to review every alert — it’s to review every alert that fired and assess its value.

The review process asks five questions about each alert that fired in the past month:

Track the outcome for each alert: keep as-is, modify threshold, update runbook, convert to ticket, or delete.

A simple review meeting agenda:

Alert Lifecycle Management

Alerts should follow a defined lifecycle from proposal to retirement.

Two signals indicate an alert should be retired:

Conclusion

Alert fatigue is a design problem, not a discipline problem. The core principle is simple: wake people up for what users experience, not for what might cause problems. High CPU doesn’t warrant a 2 AM page; elevated error rates do. Disk filling up gets a ticket; write failures get a page. Reserve the pager for symptoms, handle causes during business hours.

Derive thresholds from SLOs using multi-window burn rates that balance speed of detection with noise reduction. Every alert must have a runbook that answers: what’s the impact, how do I diagnose, how do I fix it, when do I escalate? Route alerts appropriately: pages for user-impacting issues requiring immediate action, tickets for degradation that can wait until business hours, dashboards for awareness.

Run monthly alert reviews to prune noise, update runbooks, and adjust thresholds based on real data. The goal is not zero alerts — it’s ensuring every alert that fires represents a real problem that requires human intervention, and the human receiving it has everything they need to resolve it quickly.