Node #

If the cluster is the abstraction of one big computer unit, then a Node is the individual physical machine or Virtual Machine that makes up that cluster. Nodes act as the real compute units where your application workloads run.

Understanding the internal architecture of a node, how it interacts with the control plane, how resources are managed, and how to perform safe node maintenance is a mandatory skill for platform and DevOps teams to keep clusters stable at production scale.

Two Types of Nodes in a Cluster #

Based on their responsibilities, server machines in a Kubernetes cluster are divided into two roles:

1. Control Plane Node #

A node dedicated exclusively to running the cluster’s control components (kube-apiserver, etcd, kube-scheduler, kube-controller-manager). This node must not run user application workloads so its resources stay undisturbed. In a secure production cluster, control plane nodes are marked with a special Taint so the scheduler won’t place regular application Pods there.

2. Worker Node #

A node dedicated exclusively to running real application workloads (Pods). These are the nodes with the large CPU and RAM capacity to serve user requests.

Internal Anatomy of a Worker Node #

Every Worker Node has three main components running as host OS services to manage containers:

1. Kubelet (The Main Cluster Agent) #

The kubelet is the Control Plane’s official representative on every node. It continuously watches instructions from the API Server to ensure the containers on the node run according to the requested specs.

flowchart TD
    A["Kubelet Polling the API Server"] --> B["Receive PodSpec (new/updated Pod spec)"]
    B --> C["Kubelet instructs the Container Runtime (CRI)"]
    C --> D["CRI downloads the image & runs the container"]
    D --> E["Kubelet monitors health (Liveness & Readiness Probes)"]
    E --> F["Kubelet reports node & Pod status to the API Server"]
    F --> A

The kubelet works in a closed-loop pattern (reconciliation loop). If the kubelet detects that a container inside a Pod isn’t running even though the spec says it should, it immediately tells the runtime to restart that container locally.

2. Kube-Proxy (The Node’s Network Manager) #

Kube-proxy runs on every node to maintain the cluster’s network rules. These rules enable network communication to Pods from sessions both inside and outside the cluster. Kube-proxy writes routing rules at the OS level using two main modes:

• iptables Mode: Kube-proxy writes standard Linux packet filter rules. Fast for small clusters, but performance degrades drastically when the cluster reaches thousands of Services because rule lookups are linear.
• IPVS (IP Virtual Server) Mode: Uses the Linux kernel’s internal load balancing technology. Very efficient, with more advanced load balancing algorithms (like least connection), and performance stays stable even on giant clusters with tens of thousands of Services.

3. Container Runtime (The Container Executor) #

The software responsible for managing the physical container lifecycle (downloading images, running isolated processes, stopping containers). Kubernetes doesn’t run containers directly — it does so through the standard Container Runtime Interface (CRI).

Since Docker was deprecated from Kubernetes’ internal core, the industry has migrated to leaner runtimes purpose-built for orchestration, such as containerd or CRI-O.

Node Status, Conditions, and Health #

The Kubernetes API Server periodically monitors the health of all nodes through heartbeat data sent by the kubelet. You can check basic node status with:

kubectl get nodes

To see detailed health and internal node conditions, use:

kubectl describe node <node-name>

In the detailed output, there’s a Conditions block that monitors these critical health parameters:

Pod Eviction Policy #

If a node stays in NotReady status or under resource pressure (MemoryPressure / DiskPressure) for long enough (the default tolerance timeout is 5 minutes), the Kubernetes Controller Manager takes eviction action. All Pods running on the problematic node get marked for deletion and are rescheduled by the Scheduler to another healthy node in Ready status.

Node Capacity: Capacity vs Allocatable #

A node machine has physical resource limits (CPU and RAM). However, you shouldn’t use all of that physical capacity to run application containers. If all physical RAM gets consumed by applications, the host OS will crash and take the entire node down with it.

Kubernetes divides node capacity into two categories:

1. Capacity: The total physical resources visible to the operating system.
2. Allocatable: The remaining resources that are safe and truly available for the Scheduler to allocate to application Pods.

Kubernetes calculates the Allocatable value using this formula:

Allocatable = Capacity - [Kube-Reserved] - [System-Reserved] - [Eviction-Threshold]

• Kube-Reserved: Resources reserved and locked specifically for running Kubernetes’ internal agents on the node (like Kubelet and Kube-Proxy).
• System-Reserved: Resources reserved for the host’s base operating system (like systemd, journald, SSH daemon).
• Eviction-Threshold: A critical memory margin intentionally left empty (e.g. 100Mi RAM) as a safety zone so the node has a chance to evict containers before physical RAM is truly exhausted.

The Kubernetes Scheduler only looks at Allocatable capacity when calculating whether a node can accept a new Pod based on that Pod’s resource requests.

Directing Pods: Node Scheduling Controls #

Kubernetes gives you full control over which node an application may run on:

1. Node Selector (Simple) #

Rigidly restricts Pod placement to nodes with a specific label.

# Example Node Selector Pod Manifest
spec:
  nodeSelector:
    disktype: ssd # The Pod will only run on nodes with the label disktype=ssd

2. Node Affinity (Expressive) #

A more flexible scheduling mechanism than selectors. It supports both mandatory rules (requiredDuringSchedulingIgnoredDuringExecution) and mere preferences/priorities (preferredDuringSchedulingIgnoredDuringExecution).

# Example Node Affinity Pod Manifest
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - asia-southeast1-a

3. Taints & Tolerations (Node Protection) #

Unlike selectors and affinity, which attract Pods to specific nodes, Taints push Pods away / reject them from a node. A node with a Taint will never accept any Pod, unless that Pod has a matching Toleration.

A production example: A dedicated database node with very large RAM specs gets a Taint so the developer team’s regular Frontend Pods can’t be placed there.

Node Maintenance: Cordon and Drain Safely #

When you need to perform physical server maintenance (like adding RAM, upgrading the OS kernel, or replacing hardware), you must take that node out of the cluster safely without causing service disruption for users.

Two important kubectl commands for node maintenance:

1. Cordon (Lock the Node) #

Marks the node as SchedulingDisabled. Pods already on the node keep running normally, but the Scheduler is strictly forbidden from placing new Pods on it.

# Lock the node so it won't accept new Pods
kubectl cordon node-worker-2

2. Drain (Empty the Node) #

Locks the node and also evicts and moves all Pods currently running on it to other healthy nodes.

# Empty the node for maintenance
kubectl drain node-worker-2 --ignore-daemonsets --delete-emptydir-data

After maintenance finishes, return the node to active status:

# Re-enable scheduling to the node
kubectl uncordon node-worker-2

ANTI-PATTERN: Draining a Node Running a Single-Replica Application
// WHAT WE DO:
- Run `kubectl drain node-worker-2` to upgrade the host OS kernel.
- That node turns out to be running the Payment API Pod with only 1 replica (single-replica).
// THE CONSEQUENCES IN PRODUCTION:
- The Pod is deleted from node-worker-2. While waiting for the new Pod to be created on another node
  (network initialization, image pull, JVM startup), user transaction traffic
  hits error 502 Bad Gateway. Downtime is detected in the system.

✓ THE RIGHT SOLUTION:
- Make sure all critical production applications are deployed with at least N >= 2 replicas under a Deployment.
- Configure a **Pod Disruption Budget (PDB)** to cap the maximum number of Pods that can be unavailable simultaneously during maintenance.
- Check readiness before draining: run `kubectl get pods -o wide` to spot single-replica Pods at risk of dying.

Summary #

• A node is a real compute unit — a physical server or VM where the cluster’s application containers execute.
• Key Node Components — The kubelet acts as the status-reporting agent, Kube-proxy manages the node’s local networking, and the Container Runtime executes containers.
• Capacity vs Allocatable — The Scheduler always refers to Allocatable capacity (the RAM/CPU safely left for applications after reserving a margin for Kubernetes agents and the OS).
• Scheduling Controls — Use Node Selector and Affinity to steer Pods to specific nodes, and Taints & Tolerations to protect nodes from misplaced applications.
• Safe Maintenance Procedures — Always combine cordon to lock and drain to empty a node before restarting the server to avoid downtime.

← Previous: Cluster   Next: Pod →