Logging #

In traditional monolithic architecture, tracking system errors is usually as simple as connecting an SSH terminal session to a single server and reading text log files like /var/log/nginx/access.log or /var/log/app.log using the tail command. However, in the dynamic, distributed Kubernetes ecosystem, this approach can’t be used at all. Pods are ephemeral — they can be created, rescheduled to other nodes, or destroyed at any time by the orchestrator. When a Pod dies or restarts, all local log files inside the container are lost forever. Therefore, we must design a centralized log collection system (logging pipeline) capable of collecting, enriching with cluster metadata, filtering, and storing logs from hundreds of containers in real-time and persistently.

The Three-Layer Logging Architecture in Kubernetes #

To understand how logs are processed in Kubernetes, we must view them as a three-layer data flow streaming from application containers to the centralized log database at the cluster level.

flowchart TD
    AppPod["Application Pod (Container)"] -->|"stdout / stderr"| CRI["Container Runtime (containerd)"]
    CRI -->|"Write Logs to the Host Path"| HostLog["Host Directory (/var/log/pods/)"]
    
    subgraph AgentEnrichment["Node-Level Logging Agent"]
        direction TB
        DaemonSet["Logging Agent DaemonSet (Fluent Bit / Promtail)"] -->|"Mount the Host Path"| HostLog
        DaemonSet -->|"Query metadata (pod_name, namespace)"| K8sAPI["Kube-API Server"]
        DaemonSet -->|"Enrich & Parse JSON"| ProcessLog["Log Parser & Filter Engine"]
    end
    
    ProcessLog -->|"Send Batches via Webhook / API"| Aggregators["Central Storage & UI Backend"]
    
    subgraph Aggregators
        direction LR
        Loki["Grafana Loki (Index Labels, Chunks in S3)"]
        Elasticsearch["Elasticsearch (Full-Text Indexing)"]
    end

1. First Layer: The Application (Stdout/Stderr) #

The fundamental principle of cloud-native applications (per the Twelve-Factor App methodology) is treating logs as event streams. Our applications must not manage log files independently to disk. Instead, applications must write all logs directly to the standard output channel (stdout) and standard error channel (stderr). The Container Runtime (like containerd or CRI-O) automatically captures these data streams.

2. Second Layer: Node-Level Logging #

At the host machine (node) level, the container runtime writes the log streams captured from containers to a local log file on the node disk.

• The default container log storage path is at /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<restart-count>.log.
• The system also creates symbolic links (symlinks) to the /var/log/containers/ directory to ease collection.
• These logs are stored in a simple structured format (e.g. JSON log format by the Docker daemon, or the standard CRI format).

3. Third Layer: Cluster-Level Logging #

Because node-level log files get rotated or deleted when the disk fills up, we need a cluster-level logging agent.

• This agent runs as a DaemonSet (one Pod on every worker node) like Fluent Bit or Promtail.
• The agent mounts the host /var/log/pods directory read-only.
• The agent reads new log streams, contacts the Kube-API Server to query Pod metadata (like Pod labels, namespace names, node IP addresses), inserts that metadata into log entries, and sends them in batches to the centralized log database.

Structured Logging Standardization (JSON) #

Writing logs as plain text lines is a big mistake (anti-pattern) for production-level clusters. Logs like 2026-06-17 08:00:10 INFO User login success for ID 4891 require complicated regular expressions (regex) to parse at the SIEM level, consuming lots of CPU time when processing millions of log lines per second.

We must require all applications to write logs in structured JSON format. With JSON format, every log attribute (like log level, function name, execution duration) is defined as an indexable, instantly searchable key-value pair without regex parsing processes.

Logging Format Comparison in Code #

# ANTI-PATTERN: Writing plaintext logs. Hard for automatic parsers to extract parameters.
import logging
logging.basicConfig(level=logging.INFO)
logging.info("Database connection to host %s succeeded in %d ms", "db-01.internal", 145)

# ==============================================================================
# CORRECT: Writing structured JSON logs with contextual metadata.
import sys
import json
from datetime import datetime

def log_json(level, msg, **kwargs):
    log_entry = {
        "time": datetime.utcnow().isoformat() + "Z", # ISO 8601 UTC
        "level": level,
        "msg": msg
    }
    log_entry.update(kwargs)
    sys.stdout.write(json.dumps(log_entry) + "\n")
    sys.stdout.flush()

log_json("INFO", "database_connection_established", host="db-01.internal", elapsed_ms=145)

The Standard JSON Schema for Cluster Logs #

To ease log correlation between services in different languages, we must establish a standard JSON schema that all developer teams must follow:

{
  "time": "2026-06-17T08:14:23.901Z",     // Mandatory ISO 8601 UTC format
  "level": "ERROR",                       // DEBUG, INFO, WARN, ERROR, FATAL
  "msg": "database_query_timeout",        // Concise, descriptive, snake_case
  "service": "billing-service",           // The application service name
  "version": "v1.2.4",                   // The application build version
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", // For distributed tracing correlation
  "span_id": "00f067aa0ba902b7",          // The sub-operation ID
  "elapsed_ms": 5000,                     // Performance metrics if present
  "error_details": {                      // A special object if the level is ERROR
    "class": "TimeoutException",
    "stacktrace": "at Billing.Query(db.go:45)..."
  }
}

Using the Downward API to Insert Pod Metadata #

We don’t need to force applications to manually write Pod names or node IP addresses into configuration files. We can leverage the Kubernetes Downward API to automatically inject infrastructure metadata into container environment variables, which are then read by the application logger library.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: billing-app
  template:
    metadata:
      labels:
        app: billing-app
    spec:
      containers:
      - name: app
        image: billing:v1.2.4
        env:
        # Inject the actual Pod name into an env var
        - name: K8S_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        # Inject the namespace name
        - name: K8S_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        # Inject the worker node host name
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Managing Legacy Application Logs (Legacy Sidecar Streamer) #

The challenge arises when we must migrate legacy applications whose source code can’t be modified and which hardcode log writing to files on the local disk (e.g. writing to /var/log/my-app/out.log).

If we let the application write to the container disk:

• Log files keep growing until they exhaust the container disk space, triggering node eviction due to disk pressure.
• Those logs aren’t captured by the Kubelet because they’re not sent to stdout/stderr.

Solution: The Sidecar Log Streamer Pattern #

We apply the sidecar companion container pattern, tasked with reading those log files and streaming them to the sidecar container’s stdout using the tail utility.

# Sidecar Log Streaming Implementation
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
      # 1. Main Container: Writes logs to a local file in the shared directory
      - name: legacy-main
        image: legacy-service:v4.1
        volumeMounts:
        - name: shared-log-dir
          mountPath: /var/log/app
      
      # 2. Sidecar Container: Reads the log file and streams it to stdout
      - name: log-shipper-sidecar
        image: busybox:1.36
        command: ["/bin/sh", "-c", "touch /var/log/app/out.log && tail -F /var/log/app/out.log"]
        volumeMounts:
        - name: shared-log-dir
          mountPath: /var/log/app
          
      volumes:
      # Memory-backed ephemeral volume (emptyDir) as the log storage bridge
      - name: shared-log-dir
        emptyDir:
          medium: Memory
          sizeLimit: 100Mi # Limit the capacity so it doesn't flood node RAM

With this pattern, node-level logging agents can capture the sidecar container (log-shipper-sidecar) logs because they’re streamed to stdout, while the main application keeps running without code modifications.

Log Collection Stack Comparison: EFK vs Grafana Loki #

There are two log collection architecture stacks dominating the Kubernetes ecosystem. Choosing the right technology heavily depends on operational budgets and search feature needs.

EFK and Loki Comparison Table #

Advanced Grafana Loki LogQL Query Examples #

Loki uses the LogQL language for structured log analysis. Here are query examples for solving production problems:

# 1. Show all ERROR logs from the billing service in the production namespace
{namespace="production", app="billing-service"} |= "ERROR"

# 2. Dynamically parse the JSON log payload and filter requests taking > 2 seconds
{namespace="production", app="billing-service"} 
  | json 
  | elapsed_ms > 2000 
  | line_format "Service {{.service}} is slow on path {{.path}} with duration {{.elapsed_ms}}ms"

# 3. Count the errors per minute (Rate of Errors) to create visual graphs on dashboards
sum(rate({namespace="production", app="billing-service"} |= "ERROR" [1m]))

Node Disk Protection Against Log Leaks #

If an application suffers a system loop failure (restart loop or deadlock loop) and keeps writing millions of failure log lines to stdout, the worker node host’s hard disk can fill up within minutes. This Disk Pressure condition forces the Kubelet to evict pods and can damage cluster stability.

We must configure strict log rotation at the Container Runtime (containerd) level.

containerd Log Rotation Configuration #

Create the following log rotation configuration parameters in the /etc/containerd/config.toml file on every worker node:

# containerd log rotation configuration
[plugins."io.containerd.grpc.v1.cri".containerd]
  # Limit the maximum container log file size before rotation
  # The default is often unlimited
  max_container_log_line_size = 16384 # Maximum 16KB per log line

[plugins."io.containerd.grpc.v1.cri"]
  # Automatic log rotation
  # Enable local file log rotation
  [plugins."io.containerd.grpc.v1.cri".registry]
    # ...

Note: On modern operating systems using the kubelet, log rotation is also independently managed by kubelet config parameters in /var/lib/kubelet/config.yaml.

# Add log limit parameters in the kubelet-config
containerLogMaxSize: "10Mi"       # Rotate the file if the size reaches 10 Megabytes
containerLogMaxFiles: 5           # Keep a maximum of 5 rotation backup files

With this configuration, local container log files on worker node disks never exceed 50MB (10MB x 5 files), protecting nodes from outage dangers caused by running out of disk space (disk exhaustion).

Logging Practice Anti-Patterns #

Here are bad habits operations teams and developers often do when managing logging in Kubernetes:

1. Sending Logs Synchronously Directly to the Log Database (Synchronous Remoting) #

// ANTI-PATTERN: The application calls the Elasticsearch API directly (synchronously) in the request path.
// If Elasticsearch is slow or down, the main application gets stuck/down too.
func HandlePayment(w http.ResponseWriter, r *http.Request) {
    status := ProcessPayment()
    // DON'T: The HTTP call blocks the application request thread
    http.Post("http://elasticsearch:9200/logs", "application/json", jsonLog)
    w.WriteHeader(http.StatusOK)
}

// ==============================================================================
// CORRECT: The application writes to stdout asynchronously. Let the external agent send it.
func HandlePayment(w http.ResponseWriter, r *http.Request) {
    status := ProcessPayment()
    // Just write to stdout; containerd and Fluent Bit handle the delivery to the backend
    log.Printf("{\"event\":\"payment_processed\",\"status\":\"%s\"}", status)
    w.WriteHeader(http.StatusOK)
}

2. Plain-text Secret Leaks to stdout #

Printing API authentication tokens or database passwords plainly to the container terminal output. This violates industry compliance (PCI-DSS/GDPR) because log shippers distribute them to common index databases.

3. Not Throttling/Rate-Limiting Debug Logs #

Leaving the application log level set to DEBUG in production environments. This burdens container CPU performance, floods internal cluster network bandwidth, and unnecessarily exhausts SIEM storage space. Set the minimum production log level to INFO or WARN.

Cluster Logging Practice Audit Checklist #

Use the following checklist to verify whether your logging workflow meets good production practice standards:

APPLICATION LOGGER DESIGN:
  □ Applications write all logs to stdout/stderr asynchronously.
  □ The log format uses consistent structured JSON.
  □ Log attributes include ISO 8601 UTC timestamps with millisecond precision.
  □ Log attributes include trace_id and span_id for distributed tracing.
  □ The Downward API is used to insert Pod name, namespace, and node metadata into logs.
  □ Automatic redaction (sensor) of sensitive keywords (password, credit_card) is done.

DISK & NODE-LEVEL PROTECTION:
  □ The Kubelet is configured with the 'containerLogMaxSize' limit (e.g. 10Mi).
  □ The Kubelet is configured with the maximum rotation file limit 'containerLogMaxFiles' (e.g. 5).
  □ Legacy applications writing to disk files are paired with a tail sidecar streamer.
  □ The sidecar emptyDir log volume capacity is limited using a memory-backed 'sizeLimit'.

CLUSTER-CLASS PIPELINE ARCHITECTURE:
  □ The log collection agent (Fluent Bit / Promtail) runs as a DaemonSet.
  □ The log parser is configured to automatically parse JSON logs (merge log on).
  □ Main log storage is split between hot storage (7-14 days) and cold archives (S3/GCS).
  □ Log noise levels are reduced by filtering system component health check ping status.

Summary #

• Stdout/Stderr Is the Official Log Path — Write all application logs to stdout/stderr; don’t manage log files locally on the container disk because they’re lost on restart.
• JSON Logs Ease Query Automation — Use structured JSON logging format so SIEM and visualization systems can index data efficiently without slow regex parsing.
• Limit Log Sizes for Node Protection — Configure Kubelet log rotation parameters to avoid worker node disk fill-ups from uncontrolled log loops.
• Use Sidecar Streamers for Legacy Apps — Pair legacy applications writing logs to physical files with a busybox sidecar to bypass container disk I/O limitations.
• Loki for Efficiency, EFK for Speed — Evaluate the log index architecture trade-offs; Loki significantly saves storage, while Elasticsearch excels at ad-hoc text queries.
• The Downward API Provides Infrastructure Context — Inject namespace names, Pod names, and worker nodes directly into application environment variables to enrich runtime logs.

← Previous: Security Anti-Patterns   Next: Metrics and Prometheus →