Audit Logging #

In distributed system architectures like Kubernetes, every interaction between users, applications, and internal cluster components is managed by a single component: the Kubernetes API Server (kube-apiserver). When a developer runs a kubectl command, an automated script updates the application version, or a Pod requests a Secret from etcd, all of them send HTTP requests to the API Server. In strict production environments, having full visibility into who did what, when it was done, and where the request came from is a main pillar of system security. Audit Logging provides cryptographic, structured audit trail records capturing each of those security events in sequence, offering early threat detection capabilities and invaluable forensic analysis material during security incidents.

Audit Architecture and Processing Flow #

The audit mechanism in Kubernetes runs as an integrated part of the request processing pipeline inside the kube-apiserver. Every incoming request is filtered through several evaluation stages (authentication, authorization, admission control) before finally being executed and stored in etcd. Audit log processing happens throughout this request lifecycle.

Request Stages (Audit Stages) #

To provide comprehensive visibility without constantly burdening I/O performance, the Kubernetes API Server defines four recording stages (audit stages):

Request Lifecycle Stages in the Audit Log:

1. RequestReceived:
   - Logs are recorded immediately after the API Server receives the request, but before the request is forwarded to authentication, authorization, or admission control filters.
   - Useful for detecting denial of service (DoS) attacks or tracking early request entry times.

2. ResponseStarted:
   - This stage is only triggered for long-running request types (e.g. `kubectl logs -f`, `kubectl port-forward`, or WebSocket connections).
   - Logs are recorded right after the HTTP headers are sent to the client, but before the data payload is completed.

3. ResponseComplete:
   - The final log recording stage after the entire execution process finishes and the response body is sent back to the client.
   - This is the most frequently recorded stage because it contains the final HTTP status code and the request execution result.

4. Panic:
   - A special stage triggered if an internal system failure (panic) occurs inside the API Server while processing a request.
   - Helps administrator teams investigate system bugs or anomalous inputs causing crashes.

The API Server Audit Pipeline Flow Diagram #

flowchart TD
    Client["Client Request (kubectl / ServiceAccount)"] --> K8sAPI["Kube-API Server"]
    K8sAPI --> AuthN["Authentication & Authorization"]
    
    subgraph AuditFilter["Audit Filter Engine"]
        direction TB
        RequestStage1["Stage: RequestReceived"] --> PolicyCheck{"Evaluate Policy?"}
        PolicyCheck -- "None" --> DropLog["Ignore the Log (Not Recorded)"]
        PolicyCheck -- "Metadata/Request/RequestResponse" --> WriteBackend["Send to the Log Backend"]
    end
    
    AuthN --> RequestStage1
    
    subgraph Execution["API Processing & Mutation"]
        direction TB
        MutatingAdmission["Admission Webhook (Mutating)"] --> ObjectStore["Save to etcd / Execute the Action"]
    end
    
    WriteBackend --> MutatingAdmission
    ObjectStore --> RequestStage2["Stage: ResponseComplete / Panic"]
    RequestStage2 --> WriteFinalLog["Send the Response Log to the Backend"]
    
    subgraph OutputBackends["Log Destinations"]
        direction LR
        LocalFile["Local File (/var/log/kubernetes/audit.log)"]
        Webhook["SIEM Webhook (Elasticsearch/Splunk)"]
    end
    
    WriteFinalLog --> OutputBackends

Determining Audit Detail Levels (Audit Levels) #

The biggest challenge in enabling audit logging is data volume management. If we record every request detail from all cluster components, we suffer API Server performance degradation and exponential log storage cost spikes. Therefore, we must choose the right detail levels (audit levels) for each resource type.

Audit Level Characteristics Comparison #

Production-Level Audit Policy #

The audit policy is defined in a declarative configuration file evaluated from top to bottom. The first matching rule determines the audit level for that request.

Here’s a comparison between a wrong policy configuration (data leakage) and a hardened production-level policy configuration.

# ANTI-PATTERN: Uniformly setting the audit level to RequestResponse for all resources.
# This causes Secret value leaks into local log files, and burdens
# API Server I/O by recording status requests from the kubelet and kube-proxy.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse # DON'T: This records the actual values of Secrets in logs!
  resources:
  - group: ""
    resources: ["secrets"]
- level: RequestResponse # DON'T: This records node status events flooding storage
  verbs: ["get", "list", "watch"]

# ==============================================================================
# CORRECT: A Safe, Efficient, and Informative Production-Level Audit Policy.
# File: /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived" # Ignore the early stage to reduce log size by up to 50%
rules:
  # 1. Ignore high-frequency system requests (Log Noise)
  - level: None
    users: ["system:apiserver", "system:kube-scheduler", "system:kube-controller-manager"]
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services", "services/status"]

  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get", "update", "patch"]
    resources:
      - group: ""
        resources: ["nodes", "nodes/status"]

  - level: None
    namespaces: ["kube-system"]
    resources:
      - group: ""
        resources: ["endpoints"]

  # 2. Secure Secrets and ConfigMaps (Record Metadata only, avoid recording secret values)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
    namespaces: [] # Applies to all namespaces

  # 3. Log Critical RBAC and Authentication Actions with Maximum Detail
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
      - group: ""
        resources: ["serviceaccounts"]

  # 4. Log Application Mutation Operations
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "apps"
        resources: ["deployments", "statefulsets", "daemonsets"]
      - group: ""
        resources: ["pods"]

  # 5. Default Policy (Catch-All) for User Requests
  - level: Metadata
    userGroups: ["system:authenticated"]

Enabling Audit Logging on the Control Plane #

After designing the audit policy file, we must apply it to the Kubernetes API Server. For self-managed control plane clusters (e.g. using kubeadm), the API Server runs as a Static Pod. We must edit its configuration manifest directly on the control plane server.

Configuration Steps on the Master Node #

1. Place the audit policy file at /etc/kubernetes/audit-policy.yaml on the master node host.
2. Edit the API Server manifest file /etc/kubernetes/manifests/kube-apiserver.yaml.
3. Add the audit configuration command flags:

# Excerpt from the /etc/kubernetes/manifests/kube-apiserver.yaml manifest
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    # Audit configuration flags
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30          # Log storage time range (days)
    - --audit-log-maxbackup=10       # Maximum number of stored backup files
    - --audit-log-maxsize=100        # Maximum log file size before rotation (MB)
    name: kube-apiserver
    volumeMounts:
    # Mount the audit policy file into the API Server container
    - mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit-policy
      readOnly: true
    # Mount the log storage directory on the host
    - mountPath: /var/log/kubernetes/audit
      name: audit-log-dir
  volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
  - name: audit-log-dir
    hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate

After saving the manifest, the kubelet on the master node detects the configuration change and automatically restarts the kube-apiserver Static Pod. Make sure to check whether the API Server is running normally again:

# Check the control plane pod status
kubectl get pods -n kube-system -l component=kube-apiserver

# Check whether the audit log file starts filling with JSON data
tail -n 5 /var/log/kubernetes/audit/audit.log

The Audit Log Structure (JSON Schema Analysis) #

Every entry in the audit log is recorded in a single JSON line format (JSON Lines). This makes integration with modern log parser engines easy. Here’s an example of a production-level audit log entry when a user ([email protected]) executes pod/exec:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "9c12b7a4-8b1c-4b67-827c-9b16ea9825b1",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/pods/payment-gateway-8f8f/exec?command=%2Fbin%2Fsh&container=app&stdin=true&stdout=true&tty=true",
  "verb": "create",
  "user": {
    "username": "[email protected]",
    "groups": [
      "engineering-team",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.10.45"
  ],
  "userAgent": "kubectl/v1.28.2 (darwin/arm64) kubernetes/89a41a3",
  "objectRef": {
    "resource": "pods",
    "namespace": "production",
    "name": "payment-gateway-8f8f",
    "subresource": "exec",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 101
  },
  "requestReceivedTimestamp": "2026-06-17T07:45:12.103982Z",
  "stageTimestamp": "2026-06-17T07:45:12.112489Z"
}

Key Field Explanations for Security Analysis #

• auditID: A unique UUID generated by the API Server to track the request lifecycle across all stages.
• user.username: The real identity of the request sender subject. If the request comes from an internal application, this field contains system:serviceaccount:<namespace>:<name>.
• sourceIPs: The client’s public or private IP address. Crucial for tracking token leaks from outside the office VPN network.
• objectRef.subresource: Detects special actions like exec, portforward, or attach.

Webhook and SIEM (Security Information and Event Management) Integration #

Storing audit logs on the control plane node’s local hard disk isn’t safe. If attackers gain root access to the master node, they can modify or delete the /var/log/kubernetes/audit/audit.log file to erase traces (anti-forensics).

For production-level clusters, we must send audit logs in real-time to an external log aggregator (like the Elastic Stack, Splunk, or Datadog) using webhook mode.

Audit Webhook Configuration (audit-webhook.yaml) #

We create a kubeconfig configuration file for the audit webhook server:

# File: /etc/kubernetes/audit-webhook.yaml
apiVersion: v1
kind: Config
clusters:
  - name: secure-siem-endpoint
    cluster:
      # The URL of the log collector endpoint (SIEM / Logstash / Fluentd)
      server: https://siem-collector.internal.company.com/v1/audit-receiver
      certificate-authority: /etc/ssl/certs/internal-ca.crt
users:
  - name: apiserver-audit-client
    user:
      token: "secret-bearer-token-for-auth"
contexts:
  - name: audit-context
    context:
      cluster: secure-siem-endpoint
      user: apiserver-audit-client
current-context: audit-context

Add the following flags to the kube-apiserver.yaml manifest to direct log transmission to that webhook:

- --audit-webhook-config-file=/etc/kubernetes/audit-webhook.yaml
- --audit-webhook-batch-max-size=500     # Send logs in batches of 500 lines
- --audit-webhook-batch-max-wait=10s     # Maximum transmission delay if the batch isn't full
- --audit-webhook-batch-throttle-qps=10  # Limit transmission throughput to the SIEM

Real Forensic Scenarios (Incident Investigation) #

Let’s simulate how the security team (Security Operations Center) uses audit logs to solve real security breach incidents using command-line tools (grep, jq).

Scenario A: Tracing Users Who Ran kubectl exec into Production Pods #

When there’s suspicious activity inside a database container, we want to know who has ever opened an interactive terminal (kubectl exec) into the cluster.

# Run a search for the 'exec' subresource in the audit log file
grep '"subresource":"exec"' /var/log/kubernetes/audit/audit.log | \
  jq '{
    Time: .requestReceivedTimestamp,
    User: .user.username,
    Source_IP: .sourceIPs[0],
    Namespace: .objectRef.namespace,
    Pod: .objectRef.name,
    Command: .requestURI
  }'

# The generated output:
# {
#   "Time": "2026-06-17T07:45:12.103982Z",
#   "User": "[email protected]",
#   "Source_IP": "192.168.10.45",
#   "Namespace": "production",
#   "Pod": "payment-gateway-8f8f",
#   "Command": "/api/v1/namespaces/production/pods/payment-gateway-8f8f/exec?command=%2Fbin%2Fsh..."
# }

Scenario B: Detecting Privilege Escalation #

An attacker who successfully hacks an application ServiceAccount might try creating a new ClusterRoleBinding with cluster-admin access rights to take over the entire cluster.

# Search for create/update actions against ClusterRoleBindings
grep -i "clusterrolebindings" /var/log/kubernetes/audit/audit.log | \
  grep -E '"verb":"(create|update|patch)"' | \
  jq '{
    Time: .requestReceivedTimestamp,
    Actor: .user.username,
    Action: .verb,
    TargetBinding: .objectRef.name,
    Status: .responseStatus.code
  }'

Scenario C: Detecting Mass Secret Theft (Credential Dumping) #

Secret data leaks are usually preceded by unauthorized list reads of all secrets in one namespace.

# Detect 'list' or 'watch' actions against the 'secrets' resource
grep '"resource":"secrets"' /var/log/kubernetes/audit/audit.log | \
  grep '"verb":"list"' | \
  jq 'select(.responseStatus.code == 200) | {
    Time: .requestReceivedTimestamp,
    Actor: .user.username,
    Namespace: .objectRef.namespace,
    Client: .userAgent
  }'

Audit Log Management Anti-Patterns #

Here are several implementation mistakes (anti-patterns) we must avoid in Kubernetes audit logging configuration:

1. Storing etcd Encryption Keys at the RequestResponse Log Level #

If our cluster is configured to write database encryption or sensitive data keys to the Kubernetes API, using the RequestResponse level for secret namespaces records that plain-text payload in the logs. Make sure filter rules for Secrets are always set to the Metadata level.

2. Ignoring Audit Webhook Latency #

If the SIEM webhook destination server suffers performance degradation, the API Server can block internal processing if the log queue is full.

# DON'T: Ignore the fallback policy when the webhook fails to connect.
# By default, the api-server can get stuck if the queue is full.
# CORRECT: Set the batch queue configuration dynamically with a safe buffer.
- --audit-webhook-batch-buffer-size=10000 # Temporary storage buffer
- --audit-webhook-mode=batch               # Send asynchronously (not synchronous/blocking)

3. Not Doing Time Correlation (Clock Drift) #

Audit logs are useless if the master node’s time differs from the SIEM server’s time. Make sure all master nodes actively run the Network Time Protocol (NTP) time synchronization protocol.

Production Audit Logging Checklist #

Use the following checklist to make sure your cluster’s audit logging meets industry compliance standards:

POLICY & FILTER CONFIGURATION:
  □ The audit-policy.yaml file is filtered in layers from top to bottom.
  □ System categories (kubelet, kube-proxy) are excluded with the 'None' level.
  □ Secret data (secrets, configmaps) is only recorded at the 'Metadata' level.
  □ Access right changes (RBAC, ServiceAccount) are fully recorded at the 'RequestResponse' level.
  □ The 'RequestReceived' stage is ignored to suppress log size capacity.

MASTER FILE SECURITY & ROTATION:
  □ The kube-apiserver is configured with maxage, maxbackup, and maxsize limit flags.
  □ Log directory access permissions (/var/log/kubernetes/audit) are restricted to the root user only (chmod 700).
  □ File rotation is configured using the master OS's built-in logrotate for additional compression.

SIEM & CLOUD INTEGRATION:
  □ Log transmission is sent to an external SIEM asynchronously (batch mode).
  □ Webhook credentials are securely stored with token authentication or TLS client certificates.
  □ NTP time synchronization runs actively across the entire cluster control plane.
  □ On managed cloud services (GKE/EKS), the audit logging feature is enabled in the provider console.

Summary #

• Audit Trails Are the Cluster’s Black Box — Audit logs record every activity on the Kubernetes API Server, providing absolute visibility over user and ServiceAccount actions.
• Limit Secret Logs to the Metadata Level — Don’t record Secret body contents; use the Metadata level to record who accessed them without leaking sensitive data.
• Send Logs Outside the Cluster (SIEM) — Don’t rely on local hard disks; send audit logs to a centralized analytics system (Elasticsearch/Splunk) asynchronously.
• Reduce Log Noise — Filter high-frequency system requests (like status checks from kube-proxy or the scheduler) using the None level.
• Explore with jq and grep — Fast parsing skills using JSON CLI tools accelerate incident response teams in tracing interactive kubectl exec terminals.
• Log Rotation Management Must Be Configured — Limit audit file capacity so it doesn’t flood master node storage by setting size and max backup parameters.

← Previous: Supply Chain Security   Next: Cluster Hardening →