Worker Node #

A Worker Node is the executor machine in a Kubernetes cluster. If the Control Plane is the brain designing strategy, the Worker Node is the muscle doing the physical work: running containers, managing local network traffic routing, and maintaining memory and disk allocation.

Understanding the Worker Node’s internal architecture, and how its components interact with the host OS kernel, is crucial for maintaining reliability and diagnosing application performance degradation at the production level.

Kubelet & the Node Lease API: Cluster Heartbeat Efficiency #

The kubelet is the Control Plane’s official representative on every Worker Node. Its main job is ensuring all containers on the node run according to the PodSpec sent by the API Server.

Besides watching Pod specs, the kubelet also sends heartbeat signals periodically (default every 10 seconds) to the API Server to declare that the node is still alive and healthy.

Heartbeat Evolution: The Giant Node Status Problem #

In earlier Kubernetes versions, the kubelet sent heartbeats by shipping the entire Node status data object to the API Server. This Node object was very heavy (tens of kilobytes) because it contained hardware info, kernel version, downloaded container image lists, and RAM/disk conditions.

When the cluster grew to hundreds of nodes, thousands of these heavy object requests flooded the API Server every 10 seconds, causing write congestion on the etcd database and degrading cluster performance.

The Node Lease API Solution #

Kubernetes solved this problem by releasing the Node Lease API.

Modern Heartbeat Communication:
  
  [ Kubelet ] ─── sends small data (Lease Object) every 10s ───> [ API Server ]
  [ Kubelet ] ─── sends full status (Node Object) only on changes ───> [ API Server ]

Lease Object: A tiny object (only a few bytes) acting as an attendance card. The kubelet updates the timestamp on this Lease object every 10 seconds. If etcd sees the timestamp updated, the node is considered healthy.
Node Object (Heavy): The kubelet only sends full Node status data when a significant physical change happens on the node (for example, a disk filling up or critical memory).

This approach drastically cuts API Server traffic load, allowing Kubernetes to scale clusters to thousands of nodes without performance bottlenecks.

Node Networking: Kube-Proxy Mode Comparison #

Kube-proxy is the network component running on every node. Its main job is translating Kubernetes’ abstract Service concept (a stable virtual IP address) into real network traffic routing rules inside the host machine.

Several kube-proxy working modes are commonly used in the industry:

Kube-Proxy Mode	Internal Working Method	Performance Characteristics	Main Limitations
Userspace (Legacy)	Opens a port in host userspace, intercepts TCP traffic, and routes it itself.	Very Slow (requires kernel-to-userspace context switches for every packet).	Deprecated and not recommended.
iptables	Writes routing rules using Linux’s internal firewall features (Netfilter/iptables).	Fast for small clusters. Free and stable.	Rule lookup is linear. When the cluster reaches thousands of Services, networking performance slows down.
IPVS (IP Virtual Server)	Uses the Linux kernel’s internal L4 load balancing module.	Very fast and efficient. Performance stays stable even with tens of thousands of Services.	Requires installing the dedicated IPVS kernel module on the host OS.
eBPF Bypass (Cilium)	Uses Extended Berkeley Packet Filter programs to intercept packets directly at the Linux kernel socket.	Highest performance. Bypasses all cluster iptables/IPVS overhead.	Requires a special CNI (like Cilium) and a modern Linux kernel version.

For production environments with medium-to-large microservice counts, it’s highly recommended to configure Kube-proxy in IPVS mode or adopt an eBPF-based CNI technology to minimize network overhead.

The Container Runtime Interface (CRI) in Depth #

The kubelet has no internal code for creating or stopping Docker containers. The kubelet delegates that task to an external runtime engine through an open API standard called the Container Runtime Interface (CRI).

Here’s the CRI execution flow architecture commonly used in modern production with containerd:

flowchart TD
    Kubelet["Kubelet"] -->|"gRPC (Local Domain Socket)"| CRIPlugin["containerd-cri plugin (Part of the containerd daemon)"]
    CRIPlugin --> ContainerdShim["containerd-shim (One shim process per Pod to isolate lifecycles)"]
    ContainerdShim --> Runc["runc (Low-level runtime creating Linux namespaces & cgroups)"]
    Runc --> Container["Application Container (Physically running on the host kernel)"]

Why Do We Need containerd-shim? #

The shim process acts as the container’s guardian. Once containerd finishes launching the application container through runc, containerd can be shut down or upgraded without damaging or restarting the running application containers. The shim process keeps holding the container’s file descriptors and reports their status back once containerd is active again.

Memory Allocation, Node Disk, and Eviction Policy #

Nodes have hardware resource limits. When our applications consume memory or disk close to the node’s physical limits, the Kubelet must take emergency action to prevent the host OS from crashing. This process is called Node Eviction.

The Kubelet monitors these critical (eviction thresholds) in real-time:

memory.available < 100Mi (critical remaining RAM).
nodefs.available < 10% (critical root filesystem storage).
imagefs.available < 15% (critical Docker image folder storage).

Kubelet Cleanup Order During Critical Resources: #

Garbage Collection (GC): The Kubelet first tries to clean up transient data: removing dead containers and deleting old container images no longer used by active cluster Pods.
Pod Eviction: If GC fails to free resources below the threshold, the Kubelet starts force-killing application Pods. Pods are evicted by QoS class priority (BestEffort Pods first, then Burstable, and Guaranteed last).

ANTI-PATTERN: Manually Configuring Host Node Hardware
// WHAT WE DO:
- SSH into 20 worker nodes to install log forwarding utilities (Fluentd),
  monitoring agents (Prometheus Node Exporter), or manually update host kernel configs.
// THE CONSEQUENCES IN PRODUCTION:
- Configuration Drift: Server OS conditions slowly diverge (*drift*), triggering
  anomaly bugs that are very hard to debug.
- New Nodes Unprepared: When the Autoscaler spins up new VM nodes automatically
  due to traffic spikes, those new nodes don't carry the Fluentd/Prometheus
  utilities, breaking our production monitoring pipeline.

✓ THE RIGHT SOLUTION:
- Use **DaemonSet** manifests to deploy logging and monitoring agents. A DaemonSet guarantees
  that one replica of the agent container automatically runs on every node
  (including new autoscaled nodes).
- If a container needs host kernel access (like Node Exporter), enable the
  `privileged: true` and `hostNetwork: true` features on the container spec in the YAML manifest.
- Use *Cloud-Init* scripts or automation tools (like Ansible/Terraform) for all base host node OS configuration.

Summary #

The Cluster’s Executing Muscle — Worker Nodes run real application containers and manage local node traffic routing.
Node Lease API — Reduces API Server load by replacing heavy Node status data shipments with small Lease object updates every 10 seconds for heartbeats.
Kube-Proxy IPVS Mode — Highly recommended for production because it offers much faster network rule lookup performance than standard iptables mode.
Shim & Container Lifecycle — containerd-shim acts as the container’s guardian, allowing containerd runtime updates without downtime for application containers.
DaemonSet over Manual Config — Always use DaemonSet objects to run system agents (logging/monitoring) on production nodes rather than installing them manually via SSH.

← Previous: Control Plane Next: API Server →