DaemonSet #

In an ever-growing Kubernetes cluster, there are certain infrastructure needs where every Worker Node must run the same supporting container. For example, we might want to collect logs from all containers on every node, monitor physical CPU and RAM utilization at the host machine level, or run a network routing agent (CNI plugin). If we deploy these agents with a regular Deployment, there’s no guarantee every node gets exactly one Pod. Some nodes might get two Pods while others get none. To guarantee precise one-Pod-per-node agent distribution, Kubernetes provides a special object called DaemonSet.

A DaemonSet ensures that one replica of the Pod we specify always runs consistently on every cluster node (or on every node meeting certain selection criteria). Understanding how DaemonSets work, their use of host-level access, and Control Plane toleration rules is crucial for cluster administrators to manage monitoring systems and network agents safely and efficiently.

Why Do We Need DaemonSets? (Node-Level Agents) #

Conceptually, a DaemonSet works on a very simple principle:

One Node = One DaemonSet Pod

When we deploy a DaemonSet:

1. Automatic New Node Detection: If our cluster autoscales and 3 new Worker Nodes join, the DaemonSet Controller immediately detects this and automatically schedules new DaemonSet Pods to those 3 new nodes without manual intervention.
2. Dead Node Cleanup: Conversely, if a node is removed from the cluster, the DaemonSet Controller deletes the Pod bound to that node and lets the Garbage Collector discard its leftover data.

Unlike regular Deployments, which are fully handed over to the Scheduler algorithm to place on whatever node is most empty, DaemonSets traditionally bypass most normal scheduler calculations. They directly assign the Pod to the target node using the spec.nodeName field in the internal Pod manifest.

DaemonSet Interaction with Node Cordon, Drain, and Taints #

One unique characteristic of DaemonSets is their very persistent behavior toward host node status changes:

1. Behavior When a Node Is Cordoned #

When an administrator runs kubectl cordon <node-name>, that node is marked Unschedulable. The Kubernetes Scheduler is guaranteed to never place new Pods from Deployment objects on that node. However, the DaemonSet Controller ignores this cordon restriction.

• If a new DaemonSet is deployed or a DaemonSet Pod is accidentally deleted on a cordoned node, the DaemonSet Controller still reschedules the DaemonSet Pod there.
• This is because we still need basic infrastructure services (like CNI network agents or log shippers) to stay up watching over the node, even though the node no longer accepts new application Pods.

2. Behavior When a Node Is Drained #

When we run kubectl drain <node-name>, all application Pods are force-evicted from the node so it can be safely maintained.

• By default, the kubectl drain command refuses to run and returns an error if it detects DaemonSet Pods on the node.
• To continue the drain process without deleting DaemonSets (because DaemonSets are designed to stay on nodes), we must include a special flag:

  kubectl drain <node-name> --ignore-daemonsets=true
  

Comparison Table: DaemonSet vs Static Pods #

Kubernetes has another mechanism for running Pods directly at the node level: Static Pods. Here’s a fundamental comparison to help us choose the right approach:

DaemonSet Manifest Structure in Depth #

Because DaemonSets often act as system monitoring agents or network managers, they need much higher access rights than regular applications.

Here’s a production-grade DaemonSet manifest example for monitoring physical hardware metrics (Node Exporter):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prometheus-node-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1          # Update one node sequentially
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true          # Uses the host network namespace directly
      hostPID: true              # Uses the host PID namespace to see OS processes
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        securityContext:
          readOnlyRootFilesystem: true
        ports:
        - containerPort: 9100
          hostPort: 9100         # Binds port 9100 directly to the node's physical IP
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "150m"
            memory: "128Mi"
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc            # Mounts the host's proc filesystem
      - name: sys
        hostPath:
          path: /sys             # Mounts the host's sys filesystem

Security Warning on Host Access (hostPath & hostNetwork) #

• hostNetwork: true: Lets the container use the Worker Node’s physical IP network directly. This is very fast because it avoids NAT translation overhead, but dangerous if the container port conflicts with other applications on the host OS.
• hostPath: Mounts files or directories directly from the Worker Node host’s hard disk into the container. This is mandatory for log collectors to read the /var/log/pods files.

[!CAUTION] Letting a container use hostPath: / (host root filesystem access) with the securityContext.privileged: true parameter grants full control access to the node OS. If this container gets exploited by a hacker, they can easily take over the entire physical server. Always apply read-only access (readOnly: true) to hostPath volumes in production.

Use Case 1: Node-Level Log Collectors #

In Kubernetes, container output logs (stdout/stderr) are written by the Kubelet into the local /var/log/pods folder on the Worker Node host server. To collect these logs in real-time, we place a log collector agent (like Fluent Bit or Promtail) as a DaemonSet mounting that folder.

flowchart TD
    subgraph HostNode["Worker Node Host OS"]
        LogDir["/var/log/pods/\n(Container Log Storage)"]
    end

    subgraph DaemonSetPod["DaemonSet Pod (Fluent Bit)"]
        direction TB
        HostMount["Volume Mount: /var/log"]
        Shipper["Fluent Bit Engine"]
    end

    subgraph CentralLogging["Central Infrastructure"]
        Loki["Grafana Loki / Elasticsearch"]
    end

    LogDir -->|1. Physical hostPath access| HostMount
    HostMount -->|2. Read Logs| Shipper
    Shipper -->|3. Ship logs asynchronously| Loki

    style DaemonSetPod stroke:#9b59b6,stroke-width:2px

With this model, no matter how many new application Pods are created on the node, their logs are automatically read by the same single Fluent Bit DaemonSet instance, saving RAM compared to installing a log shipper sidecar on every application Pod.

Use Case 2: Network Plugins (CNI - Container Network Interface) #

Cluster network plugins (like Cilium, Calico, or Flannel) must run on every node. Their job is manipulating Linux kernel network table rules, arranging data packet routes, and managing inter-node network traffic encryption.

• CNIs are always deployed as DaemonSets with hostNetwork: true and privileged: true configurations so they can directly create network virtual interfaces at the Worker Node’s Linux kernel OS level.
• Calico CNI, for example, needs the /run/cni, /etc/cni/net.d, and /lib/modules filesystem mounts to inject routing config files directly into the host’s network configuration folders.

Scheduling DaemonSets on Master/Control Plane Nodes #

By default, cluster master nodes (Control Plane) carry a built-in Kubernetes Taint to prevent regular applications from running there so the master node doesn’t run out of CPU:

node-role.kubernetes.io/control-plane:NoSchedule

However, metrics monitoring agents (Node Exporter) or network plugins (CNI) must also run on control plane nodes so the master node can be health-monitored and connected to the cluster network. To bypass this restriction, we must add Tolerations specs inside the DaemonSet manifest:

spec:
  template:
    spec:
      tolerations:
      # Allow Pod placement on legacy Kubernetes master nodes
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      # Allow Pod placement on modern Kubernetes control-plane nodes
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      # Allow Pod placement while the node status is not ready (maintenance)
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute

Scheduling on a Subset of Nodes #

A DaemonSet doesn’t always have to run on 100% of all cluster nodes. We can restrict placement to only node groups with certain hardware characteristics using nodeSelector or nodeAffinity label selectors.

Example use case: We want to deploy a GPU temperature monitoring DaemonSet (like the NVIDIA Device Plugin) that should only run on nodes with physical GPU graphics cards:

spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-gpu # Only run the Pod on nodes with this label

Update Strategies: RollingUpdate vs OnDelete #

Just like Deployments, DaemonSets support two container version update strategies:

1. RollingUpdate (Default): Once we apply an image update to the DaemonSet manifest, the Controller deletes the old DaemonSet Pod on one node, waits until the new Pod on that node starts healthy (Ready), then moves on to delete the Pod on the next node sequentially. Update capacity is controlled by the maxUnavailable property (usually set to 1).
2. OnDelete: When we update the DaemonSet manifest, Kubernetes doesn’t restart currently running Pods. The new update is applied to a node only if we manually delete the Pod on that node.

[!TIP] The OnDelete strategy is highly recommended when updating network plugins (CNI) or critical kernel modules in large-scale production clusters. We want to manually and gradually control when each node experiences temporary connection disruption during the network update transition to avoid mass cluster connection loss.

Anti-Patterns in DaemonSet Management #

Here are two fatal mistakes that often cause unconscious cluster resource waste:

Anti-Pattern 1: Using DaemonSets for Regular Stateless Web Applications #

Deploying business applications under DaemonSet control for routing convenience.

ANTI-PATTERN: Deploying the payment-gateway API as a DaemonSet in a 100-Node Cluster
// WHAT WE DO:
- Write a DaemonSet manifest for the payment-api application so it runs on every cluster node.
- Our cluster currently has 100 Worker Nodes.

// THE CONSEQUENCES IN PRODUCTION:
- Extreme CPU & RAM Waste: Kubernetes forces 100 payment-api application replicas to run.
- If real transaction traffic is very quiet and 3 replicas would suffice,
  we've wasted the cost allocation of 97 idle containers passively eating CPU/RAM.
- We lose the main benefit of dynamic autoscaling.

✓ THE RIGHT SOLUTION:
- Use a `Deployment` object combined with a `Horizontal Pod Autoscaler` (HPA) for stateless applications.
- Let Kubernetes adjust the replica count dynamically (e.g. scale down to 2 when quiet, scale up to 20 when busy).

Anti-Pattern 2: Ignoring Compute Resource Limits (Resource Starvation) #

Not limiting RAM and CPU allocation for DaemonSet agents running on all nodes.

ANTI-PATTERN: Injecting a Log Collector DaemonSet Without limits.memory
// WHAT WE DO:
- Deploy a Fluent Bit DaemonSet without writing the limits.memory bound in the manifest.
- A memory leak occurs inside the log shipper binary from processing corrupt logs.

// THE CONSEQUENCES IN PRODUCTION:
- Host Memory Leak: Fluent Bit's memory keeps ballooning until it consumes all physical RAM on the Worker Node.
- Because DaemonSets are main system processes, the Linux kernel detects physical RAM exhaustion
  and starts killing our critical business applications (which are in lower QoS classes) to save the node.
- The node becomes unstable, triggering cascading startup failures.

✓ THE RIGHT SOLUTION:
- Always strictly limit DaemonSet memory limits (e.g. 128Mi or 256Mi memory limits).
- Because DaemonSets run on every node, unbounded resource overhead accumulates
  massively across the cluster.

Summary #

• Automatic Node-Level Agents — A DaemonSet ensures one supporting Pod replica always runs consistently on every eligible cluster node.
• Cluster Scale Aligner — As soon as a new node joins via autoscaling, the DaemonSet automatically places a new Pod there without operator intervention.
• Ignores Cordon Restrictions — The DaemonSet Controller specifically ignores cordon status (Unschedulable) on nodes so basic infrastructure agents stay up detecting node health.
• Use hostPath & hostNetwork Carefully — Limit physical hostPath access and hostNetwork usage to minimize container security gaps.
• Tolerate the Control Plane — Write an explicit tolerations block in the manifest spec if the DaemonSet must run on control plane master nodes.
• Limit with Node Selectors — Leverage nodeSelector or nodeAffinity selectors if the DaemonSet only needs to run on nodes with special hardware (like GPU nodes).
• OnDelete for Critical Networking — Choose the OnDelete update strategy when updating network components (CNI) to avoid simultaneous cluster connection disruption.
• Resource Limits Are Mandatory — Apply strict CPU/RAM limits on DaemonSets to avoid physical host memory exhaustion (Resource Starvation).

← Previous: StatefulSet   Next: Job & CronJob →