Alerting #

Having abundant metrics and beautiful visual dashboards in Grafana won’t save your production applications if there’s no reliable alarm (alerting) system. The alerting system acts as a proactive alarm mechanism waking up the on-call engineer during real service degradation. However, designing an alerting system in Kubernetes is very challenging. Very dynamic clusters can generate thousands of noisy alarms if not configured correctly, triggering the Alert Fatigue phenomenon — a condition where operations teams get used to ignoring notifications because 95% of incoming alarms are false positives. This article explores the philosophy, implementation techniques, and hardening of alerting systems using production-level PrometheusRule and Alertmanager.

The Alert Lifecycle #

The alert data flow in Kubernetes is modularly separated between the detection engine (Prometheus) and the notification management engine (Alertmanager) to ensure message delivery reliability.

flowchart TD
    Prometheus["Prometheus Server (TSDB)"] -->|"1. Triggers Alert (Firing)"| Alertmanager["Alertmanager Engine"]
    
    subgraph DeduplicateGroup["Filtering & Grouping"]
        direction TB
        FilterInhibit{"2. Is it affected by an Inhibition Rule?"}
        FilterInhibit -- "Yes" --> SuppressAlert["Suppress the Alert (Don't Send)"]
        FilterInhibit -- "No" --> FilterSilence{"3. Is it affected by an Active Silence?"}
        FilterSilence -- "Yes" --> DropAlert["Ignore the Alert (Muted)"]
        FilterSilence -- "No" --> Grouping["4. Group (Group by alertname / namespace)"]
    end
    
    Alertmanager --> FilterInhibit
    Grouping -->|"5. Send Batches after group_wait"| RouteTree{"6. Evaluate the Routing Tree"}
    
    subgraph Receivers["Alert Receiving Destinations"]
        direction LR
        Slack["Slack Channel (#alerts-warning)"]
        PagerDuty["PagerDuty (SMS / On-Call Phone)"]
    end
    
    RouteTree -- "severity = critical" --> PagerDuty
    RouteTree -- "severity = warning" --> Slack

Workflow Stage Explanations: #

  1. Prometheus Server: Evaluates the PromQL query expressions defined in PrometheusRule. If the condition stays true for a certain time range (the for parameter), Prometheus changes the alert status to PENDING, then FIRING, then sends the alert payload to Alertmanager.
  2. Inhibition: Suppresses low-level alarms if a related high-level alarm is active.
  3. Silencing: Temporarily ignores alarms (e.g. during server maintenance schedules / maintenance windows).
  4. Grouping: Groups several similar alarms occurring simultaneously so they don’t bombard notification channels.
  5. Routing Tree: Directs alarms based on labels (e.g. severity: critical goes to PagerDuty, while severity: warning goes to Slack).

Alert Recording Policies: The PrometheusRule CRD #

To create alarm rules in Kubernetes managed by the Prometheus Operator, we use the PrometheusRule Custom Resource.

Here’s a comparison between a bad alert configuration (high noise) and hardened detection queries (actionable alerting rules).

# ANTI-PATTERN: Alarm rules triggering alert fatigue.
# Uses static CPU thresholds (CPU spikes are common)
# and doesn't set a 'for' delay, triggering alerts on every short workload burst.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: noisy-alerts
spec:
  groups:
  - name: noisy.rules
    rules:
    - alert: PodCPUHigh
      expr: container_cpu_usage_seconds_total > 0.8 # DON'T: High CPU doesn't mean the app is broken
      for: 0m # DON'T: A 1-second spike triggers an alarm
      labels:
        severity: critical # DON'T: Non-actionable alarms in the critical category

# ==============================================================================
# CORRECT: An Actionable, Structured Production-Level Alerting Configuration.
# File: prometheus-rule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-service-alerts
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  groups:
  - name: api.critical.rules
    rules:
    # 1. Alert: High Latency (Symptom-based)
    - alert: HighHttpLatencyP99
      expr: |
        histogram_quantile(0.99,
          sum by (le, job) (
            rate(http_request_duration_seconds_bucket{namespace="production"}[5m])
          )
        ) > 2.0 # P99 Latency above 2 seconds        
      for: 5m # Must stay above 2 seconds for 5 minutes
      labels:
        severity: critical
        team: platform-team
      annotations:
        summary: "HTTP P99 latency is very high on {{ $labels.job }}"
        description: "The current p99 latency is {{ $value | humanizeDuration }} (threshold limit: 2 seconds). Please investigate the down dependency."
        runbook_url: "https://wiki.company.com/runbooks/latency-triage"

    # 2. Alert: Pod Suffering CrashLoopBackOff (Infrastructure-based)
    - alert: PodCrashLooping
      expr: |
        increase(kube_pod_container_status_restarts_total{namespace="production"}[15m]) > 3        
      for: 2m
      labels:
        severity: warning
        team: devops-team
      annotations:
        summary: "Pod {{ $labels.pod }} is restarting repeatedly"
        description: "Container {{ $labels.container }} in Pod {{ $labels.pod }} has restarted {{ $value }} times in the last 15 minutes."
        runbook_url: "https://wiki.company.com/runbooks/crashloop-triage"

Why Are the Examples Above “Actionable”? #

  • runbook_url: Every alert includes a link to the investigation guide document (runbook). This ensures whoever is on duty that night immediately knows the initial investigation steps.
  • Symptom-Based: Uses user-perceived latency and application malfunction, instead of raw system metrics like CPU/RAM.

Alertmanager Routing Management Configuration #

Alertmanager manages routing, grouping, and notification receiver destinations using a central configuration file.

# File: alertmanager-secret.yaml (Kubernetes Config secret form)
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'

# Routing Tree Definition
route:
  receiver: 'slack-default-channel'
  group_by: ['alertname', 'namespace', 'job']
  group_wait: 30s        # Wait 30 seconds to collect similar alerts before sending the first batch
  group_interval: 5m     # Wait 5 minutes before sending new notifications for the same group
  repeat_interval: 12h    # Resend the same alert every 12 hours if not resolved

  routes:
  # Route A: All critical-severity alerts are directly routed to PagerDuty
  - match:
      severity: critical
    receiver: 'pagerduty-oncall'
    repeat_interval: 30m # Resend to PagerDuty every 30 minutes if not acknowledged

  # Route B: All warning-severity alerts are redirected to a secondary Slack channel
  - match:
      severity: warning
    receiver: 'slack-warnings'

# Notification Receiver Configurations
receivers:
- name: 'slack-default-channel'
  slack_configs:
  - channel: '#prod-critical-alerts'
    send_resolved: true # Send confirmation notifications if the problem is fixed
    title: '[{{ .Status | upper }}] {{ .CommonAnnotations.summary }}'
    text: >-
      *Problem Details:* {{ .CommonAnnotations.description }}
      *Namespace:* {{ .CommonLabels.namespace }}
      *Runbook:* {{ .CommonAnnotations.runbook_url }}      

- name: 'pagerduty-oncall'
  pagerduty_configs:
  - routing_key: 'pd-integration-routing-key-here'
    client: 'Kubernetes Alertmanager'
    send_resolved: true

- name: 'slack-warnings'
  slack_configs:
  - channel: '#prod-warning-alerts'
    send_resolved: true
    title: '[WARNING] {{ .CommonAnnotations.summary }}'

Silencing and Inhibition: Reducing Alert Noise #

A smart alert pipeline must be able to suppress unnecessary alarms during planned maintenance operations or when main infrastructure suffers big problems.

1. Silencing Using amtool #

amtool is the official command-line tool for interacting with the Alertmanager API. We can use it to temporarily disable alarms during maintenance schedules so they don’t disturb the on-call engineer.

# Create a 4-hour silence rule for the HighHttpLatencyP99 alert in the production namespace
amtool silence add \
  --alertmanager.url=http://alertmanager.monitoring.svc:9093 \
  alertname="HighHttpLatencyP99" \
  namespace="production" \
  --duration=4h \
  --comment="Schedule Maintenance: Database Migration" \
  --author="Platform Team"

# Example output:
# Active silence created with ID: 412b70f1-da02-47a2-9b2f-31a89c9e88ef

# View the list of currently active silences
amtool silence query --alertmanager.url=http://alertmanager.monitoring.svc:9093

# Cancel the silence rule before the duration ends
amtool silence expire 412b70f1-da02-47a2-9b2f-31a89c9e88ef --alertmanager.url=http://alertmanager.monitoring.svc:9093

2. Inhibition Rules #

Inhibition rules are used to refuse sending low-level alarms (targets) if related high-level alarms (sources) are active in the same area.

# Add inhibition rules to the Alertmanager configuration
inhibit_rules:
  # If a node is totally down (InstanceDown), suppress all disk space or kubelet api warnings for that node.
  - source_match:
      alertname: 'NodeNetworkPartitionDown'
      severity: 'critical'
    target_match:
      severity: 'warning'
    # Make sure inhibition only happens if it occurs on the identical node
    equal: ['node', 'instance']

SLO-Based Alerting #

Traditional methods setting static thresholds (like “Error rate > 5%”) are prone to two problems:

  • If traffic is quiet (e.g. 1 request per minute), 1 failure immediately produces a 100% error rate and triggers a false alarm.
  • If traffic is heavy (e.g. 10,000 requests per second), a 4.9% error rate doesn’t trigger an alarm even though it impacts thousands of users.

Instead, we must use SLO-Based Alerting monitoring the Error Budget Burn Rate.

Basic Concepts #

  • SLI (Service Level Indicator): The success request ratio, e.g. http_requests_total{status!~"5.."} / http_requests_total.
  • SLO (Service Level Objective): The reliability target, e.g. 99.9% success over a rolling 30 days.
  • Error Budget: The allowed failure tolerance: 100% - 99.9% = 0.1%.
  • Burn Rate: The Error Budget consumption speed.
    • 1x Burn Rate: The error budget will be exhausted exactly within 30 days.
    • 14.4x Burn Rate: The error budget will be exhausted within 50 hours. If this happens, we must immediately trigger a critical alert because 2% of the error budget burns in just 1 hour!

Multi-Window Multi-Burn-Rate Alerting Implementation #

To prevent detection falseness, we use two time windows simultaneously: a long window (e.g. 1 hour) for volume accuracy, and a short window (e.g. 5 minutes) to confirm the problem is still ongoing.

# SLO Alert Policy Manifest
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: billing-slo-alerts
  namespace: monitoring
spec:
  groups:
  - name: billing-slo.alerts
    rules:
    # Critical Alert: 14.4x burn rate (Critical application)
    - alert: BillingSloErrorBudgetBurnCritical
      expr: |
        (
          # Long 1-Hour Window: Error ratio > 14.4x of the 0.1% target (99.9% SLO)
          sum(rate(http_requests_total{job="billing", status_code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{job="billing"}[1h]))
          > (14.4 * 0.001)
        )
        and
        (
          # Short 5-Minute Window: Confirm errors are still ongoing right now
          sum(rate(http_requests_total{job="billing", status_code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="billing"}[5m]))
          > (14.4 * 0.001)
        )        
      for: 2m
      labels:
        severity: critical
        team: core-billing
      annotations:
        summary: "The billing Error Budget is burning too fast!"
        description: "The current error budget burn rate is 14.4x on the 1-hour and 5-minute windows. The 99.9% SLO target is threatened."
        runbook_url: "https://wiki.company.com/runbooks/slo-billing-triage"

With this method, the on-call team is only woken up if there’s real service degradation threatening our performance commitment compliance to users (SLA violations).

Alerting Anti-Patterns #

Here are fatal mistakes often found in alerting system operations:

1. Cause-Based Alerting #

# ANTI-PATTERN: Paging when container memory > 90%.
# Many applications (like Java VMs) naturally use full memory for caches
# without causing malfunctions. This produces false pages at night.
container_memory_working_set_bytes > 0.90 * container_spec_memory_limit_bytes

# ==============================================================================
# CORRECT: Paging when the application suffers a malfunction (Out Of Memory / OOM Killed).
# cAdvisor detects real container terminations.
rate(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[5m]) > 0

2. Using Email for Critical-Category Alarms #

Sending critical notifications to team emails rarely checked in real-time. For the critical category, use integrated pager alarm systems (like PagerDuty or Opsgenie) triggering phone calls to the on-call officer.

3. Alarms Without Contextual Values #

Sending alarms with minimal message content like Alert: High Latency. Without including actual values, worker node names, or the problematic pod names, investigation teams waste time manually searching for the problem source.

Production Alerting Audit Checklist #

Use the following checklist to verify your cluster’s alerting system readiness:

PROMETHEUSRULE DESIGN:
  □ All alerts are clearly classified (critical, warning, ticket).
  □ The 'for' parameter is configured to prevent noise from momentary spikes (min. 2m - 5m).
  □ Every alert includes 'summary', 'description', and 'runbook_url' annotations.
  □ Description attributes use dynamic variables ({{ $value }}, {{ $labels.pod }}).
  □ SLO-based alerting with the multi-window method is applied for critical applications.

ALERTMANAGER CONFIGURATION:
  □ Webhook credentials are securely stored in Kubernetes Secrets.
  □ Alarm grouping ('group_by') is configured based on alertname and namespace.
  □ The 'group_wait' parameter is set rationally (30s - 1m) to prevent flooding.
  □ The 'repeat_interval' parameter is set long enough (min. 4h - 12h) to avoid spamming.
  □ Inhibition rules are enabled to suppress child alarms while the parent alarm is active.
  □ Maintenance procedures include creating silence rules via the amtool CLI.

Summary #

  • Prioritize Symptom-Based Alerts — Send pages only when users experience malfunctions (high latency, high error rates); avoid alarms based on internal CPU/RAM metrics.
  • Runbook URLs Are Mandatory — Every alert must include a runbook documentation link so on-call engineers can immediately take fast triage steps.
  • Control Alert Fatigue with Time Windows — Use the for delay parameter (e.g. 5 minutes) to filter temporary fluctuations before metrics are considered dangerous anomalies.
  • Apply Inhibition to Prevent Flooding — Use Alertmanager inhibit_rules to suppress secondary warning notifications when their main supplying component is totally down.
  • amtool for Scheduled Silencing — Automate silence rule creation using the amtool CLI during cluster maintenance schedules to keep the on-call team calm.
  • SLO-Based Alerting for High Accuracy — Calculate the Error Budget Burn Rate to get precise, business-oriented warning signals.

← Previous: Metrics and Prometheus   Next: Distributed Tracing →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact