Cluster Hardening #

The security of container applications running in Kubernetes never stands alone. All forms of application-level hardening — like restricting container capabilities through SecurityContext, doing network segmentation through NetworkPolicy, or verifying image signatures in CI/CD pipelines — become useless if the cluster infrastructure underneath isn’t securely configured. If a Kubernetes API Server can be accessed without authentication, the etcd database is exposed to the public network, or the kubelet on worker nodes accepts anonymous commands, attackers can easily bypass RBAC and take full control of the entire host cluster. This article comprehensively discusses cluster hardening steps at the cluster infrastructure level to build a robust defense fortress against low-level privilege escalation threats.

Hardened Control Plane Architecture #

In a secure Kubernetes cluster, every communication path between control plane components and worker nodes must be protected with Mutual TLS (mTLS) encryption and restricted by firewalls. The following diagram illustrates the security boundary and hardened communication paths to prevent eavesdropping or command injection from the internal network.

flowchart TD
    User["Client (kubectl/Developer)"] -->|"mTLS / OIDC / Bastion"| Fire["Firewall / Load Balancer"]
    Fire -->|"Port 6443"| APIServer["kube-apiserver (Hardened)"]
    
    subgraph ControlPlane["Master Node Control Plane (Hardened)"]
        direction TB
        APIServer -->|"mTLS (Port 2379)"| etcd["etcd Database"]
        etcd -->|"KMS / Encryption at Rest"| EncryptConfig["EncryptionProviderConfig"]
        Scheduler["kube-scheduler"] -->|"mTLS"| APIServer
        ControllerManager["kube-controller-manager"] -->|"mTLS"| APIServer
    end
    
    subgraph WorkerNode["Worker Node (Hardened)"]
        direction TB
        APIServer -->|"Webhook Authentication (Port 10250)"| Kubelet["kubelet daemon"]
        Kubelet -->|"gRPC"| CRI["Container Runtime (containerd)"]
        Kubelet -->|"protectKernelDefaults: true"| Kernel["Linux Kernel (Sysctl Hardened)"]
    end

Securing the Kubernetes API Server #

The Kubernetes API Server (kube-apiserver) is the cluster’s administration center. All system modification and read actions must pass through this component. Default configurations on basic cluster installations often prioritize setup ease over security, so we must review and harden its parameter configuration.

API Server Security Configuration Flags #

Here’s a comparison between an insecure (insecure) API Server configuration and a hardened production configuration:

# ANTI-PATTERN: A very vulnerable API Server configuration.
# Allows anonymous access, uses a permissive authorization mode,
# doesn't restrict kubelet connections, and leaves the insecure (http) port active.
spec:
  containers:
  - command:
    - kube-apiserver
    - --anonymous-auth=true
    - --insecure-port=8080 # HTTP port without encryption and authentication
    - --insecure-bind-address=0.0.0.0
    - --authorization-mode=AlwaysAllow
    - --enable-admission-plugins=AlwaysAdmit

# ==============================================================================
# CORRECT: A Hardened Production API Server Configuration.
# Applies full mTLS, restricts kubelet access, and disables the HTTP path.
spec:
  containers:
  - command:
    - kube-apiserver
    - --anonymous-auth=false                   # Reject access without credentials
    - --authorization-mode=Node,RBAC           # Restrict worker node rights and use RBAC
    - --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota,LimitRanger
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 # Use strong ciphers

Key Authorization Plugin Explanations #

NodeRestriction: This admission controller plugin restricts Kubelet access rights on worker nodes. The Kubelet is only allowed to modify its own Node object and Pod objects actively scheduled to run on that node. This prevents lateral exploitation if one worker node gets hacked.
PodSecurity: Applies automatic API-level validation against Pod Security Standards natively in Kubernetes.

Securing the etcd Database #

The etcd database stores all Kubernetes cluster state data, configuration, and all application Secrets. Anyone getting direct read-write access to etcd can bypass all API Server security mechanisms, manipulatively write new RBAC objects, and take over the entire system.

1. etcd Network Isolation #

By default, the etcd process must be configured to only accept mTLS connections and must not listen on the public IP address 0.0.0.0.

# Check the etcd port socket on the Master Node
# etcd must only listen on the loopback IP (127.0.0.1) or the master node's internal private IP.
ss -tlnp | grep 2379

# A safe result (example):
# LISTEN  0  128  192.168.1.10:2379  0.0.0.0:*  users:(("etcd",pid=1234,fd=6))

2. Secret Encryption Configuration in etcd (Encryption at Rest) #

Although Secrets in Kubernetes are hidden in base64 format, that data is stored as plain text inside the etcd database. We must enable Encryption at Rest using cryptographic providers like aescbc.

# Encryption Policy Manifest: /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # The first provider in the list is used to encrypt new data.
      - aescbc:
          keys:
            - name: key1
              secret: dXNlY29uc2lnbnBhc3N3b3Jkc3VwZXJzZWNyZXQ= # Base64 of a 32-byte key
      # The identity provider is used as a fallback to read old unencrypted data.
      - identity: {}

After creating the file above, add the following flag to the kube-apiserver.yaml manifest and register its path in volumeMounts and volumes:

- --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

To verify whether encryption is successfully running, we can test by writing a new Secret and reading it directly from etcd using etcdctl:

# 1. Create a test secret
kubectl create secret generic test-secret --from-literal=token=flag{super-secret-etcd} -n default

# 2. Read the value directly from the etcd database binary using etcdctl
# If encryption succeeds, the search output in etcdctl should show random/readable 'k8s:enc:aescbc:v1'
# and must not contain the string 'flag{super-secret-etcd}' in plain text form.
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/test-secret | hexdump -C

Hardening the Kubelet Configuration #

The Kubelet is the main agent running on every worker node managing container lifecycles. By default, the kubelet port (10250) can be accessed without full authentication (anonymous access), which is an entry point for Remote Command Execution exploits.

The `/var/lib/kubelet/config.yaml` File Configuration #

We must modify the kubelet configuration on every master and worker node to disable anonymous access and enable webhook-based authorization.

# ANTI-PATTERN: A vulnerable default Kubelet configuration.
# Allows anonymous auth, the AlwaysAllow authorization mode, and opens the readonly port (10255).
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: true
authorization:
  mode: AlwaysAllow
readOnlyPort: 10255

# ==============================================================================
# CORRECT: A Hardened Kubelet Configuration for Production.
# File: /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false                   # Disable unauthenticated access
  webhook:
    enabled: true                    # Token authentication via the Kube-API Server
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook                      # Request authorization decisions from the Kube-API Server
readOnlyPort: 0                      # Disable the read-only port (10255/tcp)
protectKernelDefaults: true          # Reject kubelet startup if kernel sysctls don't match security parameters
rotateCertificates: true             # Enable automatic certificate rotation by the kubelet

After saving the configuration, restart the kubelet service on every node:

systemctl restart kubelet

Host Node OS Hardening #

Kubernetes cluster security heavily depends on the host operating system (Linux kernel) security. We must secure host kernel parameters using the sysctl module.

Kernel Hardening Parameters (`sysctl.conf`) #

Apply the following network hardening parameters in /etc/sysctl.d/99-kubernetes-security.conf on every cluster node to prevent IP spoofing routing manipulation attacks and man-in-the-middle:

# Prevent IP Spoofing by enabling Reverse Path Filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable receiving ICMP redirect packets to prevent illegal routing table modification
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Disable sending ICMP redirect packets (we don't act as a router)
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Enable protection against SYN Flood attacks (SYN cookies)
net.ipv4.tcp_syncookies = 1

# Restrict core dump file usage by regular users for memory security
fs.suid_dumpable = 0

Apply the configuration above by running:

sysctl --system

Node Firewall Configuration (Firewalld / UFW) #

We must close all node network ports and only open the minimum ports required for Kubernetes component communication.

# UFW Configuration on the Master Node (Control Plane)
ufw default deny incoming
ufw default allow outgoing

# API Server Port (Critical)
ufw allow 6443/tcp
# etcd Client Port (Only allow from other master nodes)
ufw allow proto tcp from 192.168.1.0/24 to any port 2379,2380
# Kubelet API Port
ufw allow 10250/tcp
# Kube-Scheduler & Controller-Manager Ports
ufw allow 10259/tcp
ufw allow 10257/tcp

ufw enable

Security Auditing with Kube-bench #

CIS (Center for Internet Security) publishes the standard Kubernetes security benchmark document. This document contains hundreds of specific rules for checking file permissions, module configurations, and cluster API configurations.

We can automate the CIS Kubernetes Benchmark audit using the open-source tool kube-bench from Aqua Security.

Running Kube-bench as a Kubernetes Job #

To ease compliance verification across the entire cluster, we can deploy kube-bench as a Kubernetes Job reading the control plane node spec files:

# File: kube-bench-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-control-plane
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true # Kube-bench needs host PID access to check running process parameters
      containers:
      - name: kube-bench
        image: aquasec/kube-bench:latest
        command: ["kube-bench", "run", "--targets", "master"]
        volumeMounts:
        - mountPath: /var/lib
          name: var-lib
          readOnly: true
        - mountPath: /etc
          name: etc
          readOnly: true
        - mountPath: /usr/bin
          name: usr-bin
          readOnly: true
      restartPolicy: Never
      volumes:
      - name: var-lib
        hostPath:
          path: /var/lib
      - name: etc
        hostPath:
          path: /etc
      - name: usr-bin
        hostPath:
          path: /usr/bin

Deploy the job and check the audit output logs:

kubectl apply -f kube-bench-job.yaml

# Check the audit result logs
kubectl logs -f job.batch/kube-bench-control-plane -n kube-system

The log output shows detailed assessment results (PASS, FAIL, or WARN) along with precisely recommended remediation steps:

[FAIL] 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive
[FAIL] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root

# The recommended remediation solutions given:
# Run: chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml
# Run: chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml

Patch Management and Upgrade Lifecycle #

A secure cluster today can become vulnerable tomorrow if we ignore security patch releases. The Kubernetes lifecycle is very dynamic, with new minor versions released every 4 months and old versions (more than 3 versions back) losing security backport support.

Safe Cluster Upgrade Procedures #

When doing Kubernetes version updates (e.g. from v1.28 to v1.29), we must minimize workload disruption risk by regularly draining nodes.

# 1. Drain the target node to safely evacuate all running pods to other nodes
kubectl drain worker-node-1 --ignore-daemonsets --delete-emptydir-data

# 2. Connect to the target node via SSH and upgrade the kubeadm & kubelet components
ssh worker-node-1
apt-mark unhold kubeadm kubelet
apt-get update && apt-get install -y kubeadm=1.29.0-1.1 kubelet=1.29.0-1.1
kubeadm upgrade node
systemctl daemon-reload
systemctl restart kubelet
apt-mark hold kubeadm kubelet
exit

# 3. Return the node so it can accept new pod scheduling
kubectl uncordon worker-node-1

This procedure ensures component version transitions run gradually without triggering sudden application service outages.