Metrics and Prometheus #

If logs act as individual forensic records for understanding what happened on a specific request, then metrics are aggregate data for seeing how the overall system health condition is. In large-scale Kubernetes clusters, we can’t rely on log reading to know whether the cluster is experiencing performance degradation or running out of capacity. We need numeric time-series indicators measured periodically: how many requests per second, what percentage of 5xx error statuses, what the p99 latency is, and how much node memory capacity remains. Prometheus has become the de facto industry standard for metric collection in Kubernetes because it’s specifically designed for dynamic, modular cloud-native environments.

The Prometheus Monitoring Architecture in Kubernetes #

Prometheus uses a pull-based model, where the Prometheus server actively pulls (scrapes) metrics from the target’s HTTP /metrics endpoint, instead of forcing targets to periodically push metrics. This model fits Kubernetes very well because the Kubernetes API Server can act as a dynamic directory (Service Discovery) telling Prometheus about the existence of new Pods ready to be monitored.

flowchart TD
    subgraph PullTargets["Scrape Targets (Pull-Based)"]
        direction TB
        AppPod["App Pods (Instrumented with SDK, Port 9090 /metrics)"]
        cAdvisor["cAdvisor (Kubelet Port 10250)"]
        KSM["kube-state-metrics (State of K8s Objects)"]
        NodeExporter["node-exporter (Node Port 9100)"]
    end
    
    PrometheusServer["Prometheus Server (TSDB Engine)"] -->|"Scrape (Port 9090)"| PullTargets
    
    subgraph OperatorConfig["Prometheus Operator Config"]
        direction TB
        ServiceMonitor["ServiceMonitor / PodMonitor (CRD)"]
        PrometheusRule["PrometheusRule (Alert & Recording Rules)"]
    end
    
    PrometheusServer -.->|"Discover Targets via"| ServiceMonitor
    PrometheusServer -->|"Evaluate Rules"| PrometheusRule
    
    PrometheusServer -->|"Send Alerts"| Alertmanager["Alertmanager"]
    Alertmanager -->|"Push Alerts"| SlackPager["Alert Channels (Slack / PagerDuty)"]
    
    Grafana["Grafana Dashboard"] -->|"PromQL Queries"| PrometheusServer

Prometheus Ecosystem Components #

  1. Prometheus Server: The system’s heart executing scraping, storing data in the local time-series database (TSDB), and evaluating PromQL rules.
  2. Prometheus Operator: A Kubernetes operator automating Prometheus deployments and allowing target configuration using Custom Resource Definitions (CRDs) like ServiceMonitor and PodMonitor.
  3. Kube-State-Metrics (KSM): An internal monitoring agent listening to Kubernetes object API changes (like Deployment, Pod, PVC status) and exposing them as numeric metrics.
  4. Node-Exporter: An OS-level agent running on every node to expose hardware metrics (CPU usage, disk I/O, host network bandwidth).
  5. cAdvisor: A utility embedded directly inside the Kubelet to monitor container performance metrics at the cgroup level (container memory usage, CPU throttling).

Understanding Prometheus Metric Types #

To write correct PromQL queries, we must understand the four basic metric types sent by Prometheus client libraries:

Key Prometheus Metric Types:

1. Counter (Cumulative, Only Increases):
   - A numeric value that can only increase (monotonic) or return to 0 when the app restarts.
   - Must end with the '_total' suffix in the name.
   - Examples: http_requests_total, process_cpu_seconds_total.
   - Rule: Always use the rate() or increase() function when querying Counters.

2. Gauge (Up-Down Values):
   - An instant numeric value that can freely go up or down.
   - Examples: memory_usage_bytes, active_threads, replica_count.
   - Rule: Never use the rate() function on Gauge metrics.

3. Histogram (Frequency Distribution):
   - Measures the distribution of events (like request latency or payload sizes) into predetermined numeric buckets.
   - Produces three automatic time-series:
     - '<metric_name>_bucket{le="<upper_limit>"}' (the event count below the le limit)
     - '<metric_name>_sum' (the total sum of measured values)
     - '<metric_name>_count' (the total event count)
   - Rule: Use the histogram_quantile() function to calculate percentile values (p95, p99).

4. Summary (Client-Side Quantiles):
   - Similar to a Histogram, but percentiles (quantiles) are calculated directly on the client application side before being sent to Prometheus.
   - Much lighter on Prometheus server CPU load, but Summary data can't be mathematically aggregated across different pods/instances.

Application Instrumentation Using Official SDKs #

We must avoid writing metric formats manually (plaintext manual serialization). Always use the official Prometheus client SDK to register metrics globally, ensuring the /metrics output format complies with the OpenMetrics standard.

Here’s a comparison between an unsafe metric instrumentation handling approach (manual synchronous) and the standard approach using official SDKs in Go and Python.

// ANTI-PATTERN: Writing the metrics endpoint manually by formatting strings synchronously.
// This approach is prone to format errors and burdens application memory processing.
func MetricsHandler(w http.ResponseWriter, r *http.Request) {
    // DON'T: Doing calculations manually on the HTTP thread
    fmt.Fprintf(w, "http_requests_total{status=\"200\"} %d\n", requestCount)
}

// ==============================================================================
// CORRECT: Using the official prometheus/client_golang SDK.
// Metrics are registered once globally, and are automatically thread-safe.
package main

import (
    "net/http"
    "time"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    // Counter registration with dimension labels
    httpRequestsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "The total number of processed HTTP requests.",
        },
        []string{"method", "endpoint", "status_code"},
    )
    
    // Histogram registration for Latency
    httpRequestDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "The HTTP request processing duration in seconds.",
            Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0},
        },
        []string{"method", "endpoint"},
    )
)

func InstrumentMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        
        // Wrap the ResponseWriter to capture the status code (e.g. 200, 500)
        wrappedWriter := NewResponseWriterWrapper(w)
        
        next.ServeHTTP(wrappedWriter, r)
        
        duration := time.Since(start).Seconds()
        
        // Record metrics safely and asynchronously
        httpRequestsTotal.WithLabelValues(r.Method, r.URL.Path, wrappedWriter.StatusString()).Inc()
        httpRequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)
    })
}

func main() {
    // Expose the SDK's built-in /metrics endpoint
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":9090", nil)
}

Scraping Target Configuration with ServiceMonitor #

If we use the Prometheus Operator, we don’t need to manually edit the prometheus.yml configuration file and restart the Prometheus server when new applications appear. We just deploy a ServiceMonitor manifest at the application namespace level.

The Prometheus Operator watches for the existence of these Custom Resource files, dynamically reconfigures the scraping targets, and applies them to the Prometheus Server instantly.

# Scraping Target Definition: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: billing-service-monitor
  namespace: monitoring       # The namespace where the Prometheus Operator runs
  labels:
    release: prometheus-stack # The matching label so the Prometheus instance recognizes it
spec:
  namespaceSelector:
    matchNames:
    - production              # Watch the target namespace
  selector:
    matchLabels:
      app: billing-service    # Find Services with this label
  endpoints:
  - port: metrics-port        # The name of the exposed Service port
    path: /metrics
    interval: 15s             # Scrape evaluation every 15 seconds
    scrapeTimeout: 10s        # Timeout if the endpoint responds slowly

For the ServiceMonitor above to detect target pods, we must configure the application Service with the matching port name:

# The Exposed Application Service
apiVersion: v1
kind: Service
metadata:
  name: billing-service
  namespace: production
  labels:
    app: billing-service
spec:
  selector:
    app: billing-app
  ports:
  - name: http-port
    port: 80
    targetPort: 8080
  - name: metrics-port        # Must match the port declaration in the ServiceMonitor
    port: 9090
    targetPort: 9090

Writing Advanced PromQL Queries (PromQL in Production) #

PromQL (Prometheus Query Language) is a very powerful query language but requires precise mathematical understanding, especially when dealing with Histograms.

1. Calculating the 99th Percentile Latency (p99 Latency) #

To calculate the maximum latency experienced by 99% of our application users, we must combine the Histogram metric with the following query:

# p99 Latency Query per Endpoint
histogram_quantile(0.99,
  sum by (le, endpoint) (
    rate(http_request_duration_seconds_bucket{namespace="production"}[5m])
  )
)

[!CAUTION] Common Mistake Warning: We must use the sum by (le, ...) operator before the histogram_quantile() query. If we ignore the le (less or equal) label in the sum clause, the quantile calculation loses its bucket reference and produces mathematically wrong values or NaN.

2. Measuring the 5xx Error Status Percentage (Error Rate %) #

Looking at absolute error counts (e.g. 10 errors) isn’t informative if we don’t compare them with total traffic. We must calculate the error percentage ratio:

sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

3. Detecting Container CPU Throttling #

Sometimes an application runs slowly even though its average CPU utilization looks low. This is usually caused by CPU throttling due to overly tight limit boundaries.

# Calculate the CPU throttling rate in seconds per Pod
sum by (pod) (
  rate(container_cpu_cfs_throttled_seconds_total{namespace="production", container!=""}[5m])
)

Recording Rules for Dashboard Performance Optimization #

When we have a Grafana dashboard containing dozens of complex PromQL graphs (e.g. calculating p99 latency over millions of time-series), the Prometheus Server must massively read disk data every time the dashboard refreshes. This can cause memory exhaustion (Out Of Memory / OOM) on the Prometheus server.

To solve this, we must apply Recording Rules to periodically compute those complex queries in the background and store the results into new lightweight metric names.

# Recording Rules Definition: PrometheusRule
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: billing-recording-rules
  namespace: monitoring
spec:
  groups:
  - name: billing-performance.rules
    interval: 30s # Compute the rules every 30 seconds
    rules:
    # 1. Pre-compute the request rate per second
    - record: job:http_requests:rate5m
      expr: sum by (job, method) (rate(http_requests_total[5m]))
      
    # 2. Pre-compute the p99 latency
    - record: job:http_request_duration_p99:rate5m
      expr: histogram_quantile(0.99, sum by (job, le) (rate(httpRequestDuration_bucket[5m])))

At the Grafana Dashboard level, we just query the new metric name job:http_requests:rate5m, which runs very fast because it’s instant data without dynamic aggregation operations.

The Four Golden Signals #

Per Google Site Reliability Engineering (SRE) guidance, here are the four main signals that must be monitored and dashboarded for every production application service:

The Four Golden Monitoring Signals:

1. Latency:
   - The time needed to serve one request.
   - p99 query: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

2. Traffic (Throughput):
   - The request size or load volume on the network.
   - Query: sum(rate(http_requests_total[5m])) by (job)

3. Errors:
   - The rate of failed requests, measured as a percentage or ratio.
   - Query: sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

4. Saturation:
   - A measure of how "full" your system resources are.
   - Memory Query: container_memory_working_set_bytes / container_spec_memory_limit_bytes * 100

Metric Security and Performance Anti-Patterns #

Avoid the following common mistakes in metric-based monitoring operations:

1. High Cardinality #

This is Prometheus’s number one memory killer. High cardinality happens when we insert unique label values for every request, like user_id, email, transaction_uuid, or ip_address into metrics.

// ANTI-PATTERN: Including unique transaction IDs into metric labels.
// This creates millions of time-series in the TSDB database and triggers OOM Crashes!
httpRequestsTotal.WithLabelValues("GET", "/checkout", "200", transactionUUID).Inc()

// ==============================================================================
// CORRECT: Labels are only filled with categorical data with limited values (low cardinality).
httpRequestsTotal.WithLabelValues("GET", "/checkout", "200").Inc()

2. Ignoring Scraping Timeouts and Intervals #

Setting scrape intervals too short (e.g. < 1s) on Java applications with slow startup times. This triggers internal processing queues (scrape overlap) increasing container runtime latency. Use rational intervals between 15s and 30s.

Production Metric Monitoring Checklist #

Make sure all the following monitoring criteria are met before releasing services to production:

METRICS & CODE INSTRUMENTATION:
  □ Applications are instrumented using the official Prometheus SDK.
  □ Metric naming complies with standard conventions (e.g. Counters end with '_total').
  □ All metric labels have low cardinality (no UIDs, emails, UUIDs).
  □ Histogram metrics are configured with bucket limits matching the application characteristics.
  □ The '/metrics' endpoint is protected from outside public access using internal authentication or Ingress restrictions.

KUBERNETES RUNTIME CONFIGURATION:
  □ 'ServiceMonitor' or 'PodMonitor' files are created for every microservice.
  □ The metrics port on the application Service is defined with a clear name (e.g. 'metrics-port').
  □ Kube-State-Metrics (KSM) is active to monitor replica status and pod stability.
  □ cAdvisor is active to monitor container memory limit usage.

QUERIES & OPTIMIZATION:
  □ The main dashboard visualizes the 'Four Golden Signals' centrally.
  □ PromQL histogram_quantile queries include the 'sum by (le)' clause.
  □ Recording Rules are configured for high-computation-load dashboard queries.

Summary #

  • The Pull Model Eases Auto-Discovery — Prometheus’s pull-based metric model simplifies target configuration; Prometheus automatically discovers new Pods through Service Discovery integration.
  • High Cardinality Kills Memory — Keep metric labels categorical; inserting unique data like user IDs or UUIDs causes the Prometheus server to OOM crash.
  • Recording Rules Speed Up Grafana — Use PrometheusRule recording rules to periodically compute heavy queries, keeping Grafana dashboards responsive when opened.
  • Histogram Quantiles Must Use le — When calculating latency percentiles (p95/p99), make sure your sum operator doesn’t discard the le label to avoid math errors.
  • kube-state-metrics for Object Visibility — Leverage KSM built-in metrics to detect application pod replica deviations without writing manual monitoring code.
  • Monitor the Four Golden Signals — Make the Latency, Traffic, Errors, and Saturation metrics the basic foundation of every production service monitoring dashboard.

← Previous: Logging   Next: Alerting →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact