Grafana Dashboard #

Collections of numeric metrics pulled by Prometheus from every corner of the cluster won’t provide operational value if they can’t be visualized quickly, clearly, and relevantly during incidents. In crisis situations where production systems fail, on-call engineer teams must not waste time writing PromQL queries manually in raw consoles to find problem sources. We need integrated visual dashboards capable of presenting system health instantly. Grafana has become the industry standard for metric visualization because of its ability to connect various data sources and present dynamic dashboards that cut problem identification time (Mean Time to Detect / MTTD) from hours to minutes.

Monitoring Dashboard Hierarchy (Dashboard Drill-down Pattern) #

One of the biggest dashboard design mistakes is putting all metric graphs (from master node CPU temperatures to microservice database query latency) into one single, very long dashboard page. This burdens the accessing browser’s memory, slows down Prometheus queries, and floods human cognitive focus with irrelevant information.

For production operations, we must apply the Hierarchical Drill-down pattern divided into four separate dashboard levels:

flowchart TD
    StartView["1. Level 1 Dashboard: Global Cluster Overview"] -->|"Detect Node / Namespace Anomalies"| NamespaceView["2. Level 2 Dashboard: Namespace Overview"]
    NamespaceView -->|"Find the Problematic Service (High Error/Latency)"| ServiceView["3. Level 3 Dashboard: Service Detail (Pods & DB)"]
    ServiceView -->|"Click the Graph Anomaly Point (Exemplars)"| TraceView["4. Level 4 Dashboard: Tracing Span (Grafana Tempo)"]
    
    subgraph GoldenSignals["Main Metrics at Every Level"]
        direction LR
        Throughput["Traffic (Volume)"]
        Latency["Latency (Response Time)"]
        Errors["Errors (Error Rate %)"]
        Saturation["Saturation (Resource Limit %)"]
    end
    
    NamespaceView -.-> GoldenSignals
    ServiceView -.-> GoldenSignals

Dashboard Level Explanations: #

  1. Level 1: Global Cluster Overview: A high-level dashboard for monitoring the cluster’s physical capacity (total CPU, memory, ready worker node status, pod restart / OOM killed incidents). Meant for SRE and infrastructure teams.
  2. Level 2: Namespace Overview: Presents comparison visualizations between applications in one namespace. Visualizes the Four Golden Signals concisely to detect which application suffers anomalies.
  3. Level 3: Service Detail: A deep dashboard for diagnosing one specific application. Shows internal runtime metrics (like GC pauses, thread counts, specific database query latency, and external connection status).
  4. Level 4: Tracing details: Individual request tracking (Trace IDs) to find slow code lines or network dependencies.

Four Golden Signals Visualization (Four Golden Signals Panels) #

Every application-level dashboard (Levels 2 & 3) must present panels for monitoring the four golden signals. Here are the PromQL configurations and optimal visualizations for each panel:

1. Latency #

  • PromQL Query (p99 & p50):
    # p99 Latency (Critical Threshold)
histogram_quantile(0.99, sum by (le, job) (rate(http_request_duration_seconds_bucket{namespace="$namespace", job="$service"}[5m])))
  • Visualization Type: Time Series.
  • Unit: seconds (s) or milliseconds (ms).
  • Thresholds: 0s - 0.5s (Green), 0.5s - 1.5s (Yellow), >1.5s (Red).

2. Traffic (Throughput) #

  • PromQL Query:
    sum(rate(http_requests_total{namespace="$namespace", job="$service"}[5m])) by (status_code)
  • Visualization Type: Time Series with a stacked area chart based on status codes (2xx statuses green, 4xx blue, 5xx red).
  • Unit: requests per second (ops).

3. Errors (Error Rate) #

  • PromQL Query (Ratio %):
    sum(rate(http_requests_total{namespace="$namespace", job="$service", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{namespace="$namespace", job="$service"}[5m])) * 100
  • Visualization Type: Stat (Single Stat) to show a big instant percentage number.
  • Unit: percent (0-100).
  • Thresholds: 0% - 0.1% (Green), 0.1% - 1% (Yellow), >1% (Red).

4. Saturation #

  • PromQL Query (CPU & Memory % against Limits):
    # Actual memory utilization against the Kubernetes memory limit
sum(container_memory_working_set_bytes{namespace="$namespace", container!=""}) by (pod)
/
sum(container_spec_memory_limit_bytes{namespace="$namespace", container!=""}) by (pod) * 100
  • Visualization Type: Gauge (or Bar Gauge).
  • Unit: percent (0-100).
  • Thresholds: 0% - 70% (Green), 70% - 85% (Yellow), >85% (Red).

Templating and Dynamic Variables (Dynamic Dashboards) #

We must avoid creating separate dashboards for every application (e.g. dedicated dashboards for billing-app-dev, billing-app-prod, inventory-app). This habit triggers an uncontrolled dashboard sprawl explosion and complicates layout updates.

Instead, use Template Variables on Grafana dashboards. These variables generate dynamic dropdown menus at the dashboard top for filtering visualizations by datasource, namespace, and service name.

Here’s the recommended JSON variable configuration on the dashboard settings:

{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "label": "Data Source",
        "hide": 0
      },
      {
        "name": "namespace",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(kube_namespace_labels, namespace)",
        "label": "Namespace",
        "multi": false,
        "includeAll": false,
        "refresh": 1
      },
      {
        "name": "service",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(http_requests_total{namespace=\"$namespace\"}, job)",
        "label": "Service Name",
        "multi": false,
        "includeAll": false,
        "refresh": 2
      }
    ]
  }
}

These $namespace and $service variables are then directly referenced inside dashboard panel PromQL queries, like http_requests_total{namespace="$namespace", job="$service"}.

Dashboard-as-Code Implementation via GitOps #

Creating and editing dashboards directly using the Grafana graphical interface (UI) is a dangerous anti-pattern in production environments. If dashboards are manually modified:

  • We lose the change history trail (version control).
  • Dashboards are prone to accidental deletion or damage by other developers.
  • Dashboard configuration differences (config drift) occur between clusters.

We must manage dashboards as code (Dashboard-as-Code) by storing them as JSON files in Git repositories, then deploying them to clusters using ConfigMap objects watched by Grafana auto-provisioning agents.

Dashboard Provisioning Manifest via ConfigMap #

If we deploy Grafana using the kube-prometheus-stack Helm chart, there’s a k8s-sidecar sidecar container actively looking for ConfigMaps with special labels and automatically converting them into runtime Grafana dashboards.

# File: billing-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-billing-dashboard
  namespace: monitoring
  labels:
    # This label is mandatory for detection by the Grafana sidecar container
    grafana_dashboard: "1"
spec:
  # Optional configuration to separate dashboard categories into Grafana folders
  annotations:
    grafana_dashboard_folder: "Application Performance"
data:
  # The dashboard JSON file name
  billing-service-overview.json: |
    {
      "annotations": {
        "list": []
      },
      "editable": false,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "id": null,
      "links": [],
      "liveNow": false,
      "panels": [
        {
          "type": "timeseries",
          "title": "Throughput billing-service",
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "targets": [
            {
              "datasource": "${datasource}",
              "expr": "sum(rate(http_requests_total{namespace=\"$namespace\", job=\"$service\"}[5m])) by (status_code)",
              "refId": "A"
            }
          ]
        }
      ],
      "schemaVersion": 38,
      "style": "dark",
      "tags": ["application", "production"],
      "title": "Billing Service Overview",
      "uid": "billing-app-overview",
      "version": 1
    }

Metric-to-Trace Interconnection Using Exemplars #

Seeing a sharply rising p99 latency graph tells us that there’s a problem, but doesn’t tell us which individual request is slow. Exemplars is a Prometheus and Grafana feature binding distributed tracing Trace IDs directly to specific data points on metric graphs.

When Exemplars are configured:

  1. The Grafana dashboard shows small star points inside latency time series charts.
  2. When the cursor hovers over those points, Trace ID details appear.
  3. We can click that Trace ID to instantly open the detailed span visualization (Grafana Tempo) on the dashboard’s right side.
Exemplars Flow Visualization:

[Latency Time Series Graph] ──> There's a High Latency Anomaly (3.5s)
                                       │
                                       ▼ (Hover the Cursor over the Exemplar Star)
                               [Pop-Up Trace ID: 4bf92f3577b3...]
                                       │
                                       ▼ (Click the Trace ID Link)
                        [Direct Drill Down to the Grafana Tempo Panel]

Panel Configuration with Exemplars Support #

Inside the Grafana dashboard JSON manifest, we enable exemplars on the panel query target options:

"targets": [
  {
    "datasource": "${datasource}",
    "expr": "histogram_quantile(0.99, sum by (le, job) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\"}[5m])))",
    "refId": "A",
    "exemplar": true # Enable exemplar visualization
  }
]

Note: Make sure your cluster’s Prometheus database is configured with the --enable-feature=exemplars-storage flag at startup.

Installing Alert Annotations on Dashboards #

To ease incident correlation analysis, we can show active alarm moments (alert firing) directly as red vertical lines inside Grafana dashboards.

Add the following annotations configuration at the top level of your dashboard JSON structure:

"annotations": {
  "list": [
    {
      "builtIn": 1,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "enable": true,
      "hide": false,
      "name": "Alerts Firing Events",
      "type": "dashboard",
      "iconColor": "rgba(255, 96, 96, 1)",
      "target": {
        "limit": 100,
        "matchAny": false,
        "tags": ["production", "alert"]
      }
    }
  ]
}

When a PrometheusRule switches to the firing status, Grafana plots red dashed lines on the graph area, showing instant visual correlation between traffic spikes and triggered alarms.

Dashboard Design Anti-Patterns #

Avoid the following fatal mistakes when designing visualization interfaces:

1. The “Wall of Charts” Pattern (Too Many Graphs) #

Placing more than 20 graph panels on one dashboard page. This degrades user browser render performance and burdens the Prometheus server with dozens of concurrent queries.

✓ SOLUTION: 
- Limit the main dashboard to a maximum of 8-12 important panels.
- Use the "Collapsible Rows" feature to hide secondary metrics by default.

2. PromQL Queries Without Appropriate Measurement Units #

Showing memory file size byte data without setting the Unit parameter in Grafana panel settings. Numbers like 8589934592 are very hard for humans to read instantly compared to setting the unit to Bytes (IEC), which automatically formats it into 8 GiB.

3. Manual Dashboard Duplication (Manual Dashboard Sprawl) #

Duplicating the same dashboard for every new microservice. Use dynamic variable templating to create one reusable template dashboard.

Production Grafana Dashboard Audit Checklist #

Make sure your visualization dashboards meet the following quality standards before being used by operations teams:

DISPLAY & STRUCTURE DESIGN:
  □ Dashboards follow the hierarchical drill-down pattern (Cluster -> Namespace -> Service).
  □ Dashboard panels include 'Four Golden Signals' visualizations for critical applications.
  □ Collapsible rows are used to hide secondary metric panels.
  □ All measurement units are precisely configured (bytes, seconds, percent).
  □ Standard color thresholds (green, yellow, red) representing real urgency are used.

DYNAMICS & VARIABLE TEMPLATING:
  □ The '$datasource' variable is used for cluster switching flexibility.
  □ The '$namespace' and '$service' dropdowns are active and PromQL queries are interlinked.
  □ No hardcoded values for cluster, namespace, or pod parameters exist in PromQL queries.

INTEGRATION & PROVISIONING (GITOPS):
  □ Dashboards are stored as JSON files in Git repositories.
  □ Dashboards are automatically provisioned to clusters using ConfigMap objects with matching labels.
  □ Dashboards are locked (editable: false) at the JSON level so they can't be manually modified in production.
  □ The alert annotations feature is enabled on time series charts.
  □ Exemplars are configured to connect latency graphs directly to Grafana Tempo traces.

Summary #

  • Reduce MTTD with Hierarchical Design — Separate dashboards into Cluster, Namespace, and Service Detail levels to ease gradual incident investigations.
  • Mandate Four Golden Signals Visualization — Make sure the Latency, Traffic, Errors, and Saturation metrics are always the main visual focus of your application dashboards.
  • Use Variables for Reusable Dashboards — Leverage the $namespace and $service variables to prevent manual dashboard duplication triggering file management chaos.
  • Apply Dashboard-as-Code via GitOps — Don’t manually edit dashboards in production; manage them as JSON in Git repositories and deploy using ConfigMaps.
  • Connect Metrics to Tracing via Exemplars — Enable the exemplars feature to instantly jump from latency anomaly graphs to tracing visualizations in Grafana Tempo.
  • Configure Measurement Units Accurately — Set visualization units (like changing seconds to milliseconds or bytes to GiB) so data can be understood by on-call teams within 1 second.

← Previous: Distributed Tracing   Next: Health Check →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact