CNI Plugin #

As the most popular container orchestration system, Kubernetes has a design philosophy that strongly emphasizes modularity. Kubernetes establishes strict network contracts (like Flat Networks and unique per-Pod IP allocation), but the Kubernetes API Server itself doesn’t process virtual network device creation, routing tables, or packet encapsulation rules. Those low-level operational tasks (low-level data path) are fully delegated to Container Network Interface (CNI) Plugins.

CNI is a minimalist spec standard defining how container runtimes (like containerd or CRI-O) interact with third-party network plugins through standard JSON configuration manifest files.

Choosing a CNI plugin at cluster creation time is one of the most critical architectural decisions we must make. The CNI we choose determines our cluster’s network I/O performance, security feature availability (internal firewalls via NetworkPolicy), inter-node data traffic encryption capacity, and system observability ease.

This article dissects how CNI works under the hood, compares the most popular CNI plugins (Flannel, Calico, and Cilium), and outlines CNI migration procedures on production clusters.

How CNI Works: The Pod Creation Lifecycle #

When we deploy a new Pod in the cluster, the Kubelet on the worker node doesn’t directly know how to give that Pod an IP. The Kubelet relies on the CNI plugin binary installed on the worker node host to handle that task.

Physically, CNI configuration files live in the host directory /etc/cni/net.d/, while the executable binaries live in /opt/cni/bin/.

Let’s look at how this Pod network creation coordination flow happens through the following diagram:

flowchart TD
    API["Kubernetes API Server"] -- "Create Pod Instruction" --> Kubelet["Kubelet (Worker Node)"]
    Kubelet -- "CRI Request (Create Sandbox)" --> CRI["Container Runtime (containerd)"]
    CRI -- "CNI ADD Command" --> CNI["CNI Plugin Binary (/opt/cni/bin/)"]
    
    subgraph CNIOperations["Internal CNI Plugin Operations"]
        veth["1. Create a virtual veth pair"]
        IPAM["2. Take an IP from the Pod CIDR (IPAM)"]
        netns["3. Insert eth0 into the Pod netns"]
        Route["4. Set up the host default gateway"]
        Firewall["5. Apply NetworkPolicy rules"]
    end
    
    CNI --> CNIOperations
    CNIOperations -- "Return IP & MAC" --> CRI
    CRI --> Kubelet

1. The ADD Command (When a Pod Is Created) #

As soon as the Kubelet receives instructions from the API Server to start a Pod on the worker node:

1. The Kubelet calls the Container Runtime Interface (CRI, e.g. containerd) to create a sandbox container.
2. containerd reads the /etc/cni/net.d/ directory to learn which CNI plugin is active, then executes that CNI binary with the ADD command argument.
3. The CNI binary creates a pair of virtual network cards (veth pair), inserts one end (named eth0) into the Pod’s isolated network namespace, and plugs the other end into the host virtual bridge (like cbr0 or cni0).
4. The CNI interacts with the IPAM (IP Address Management) plugin to take one available free IP from that worker node’s CIDR block, then assigns it to the Pod’s eth0 interface.
5. The CNI writes routing rules into the worker node host’s IP table so all nodes know where to send packets when reaching that new Pod IP.
6. Finally, the CNI writes iptables/eBPF rules to apply NetworkPolicy restrictions if configured.

2. The DEL Command (When a Pod Is Deleted) #

When a Pod is shut down, containerd calls the CNI binary with the DEL command. The CNI removes the created virtual interfaces, releases the IP back to the IPAM pool for future Pods, and cleans up all related routing and firewall rules on the worker node host.

In-Depth Analysis of the Most Popular CNI Plugins #

Every CNI plugin is designed with different philosophies, architectures, and use cases. Let’s dissect the three CNI giants most used in the industry today:

1. Flannel: Simple and Minimalist #

Flannel is one of the oldest and simplest CNIs, developed by CoreOS. Flannel’s philosophy is very simple: provide a flat network between worker nodes as fast and easily as possible.

Flannel Architecture (Default VXLAN Encapsulation):
  [Pod A (10.244.1.5)] ──> [cbr0 Bridge] ──> [flannel.1 Tunnel] ──> UDP Encapsulation ──> Physical Network Card

• Routing Mechanism: By default, Flannel wraps original Pod packets into UDP packets using VXLAN technology (overlay network). Flannel also supports host-gw mode (host gateway) that sends packets directly without encapsulation, but this mode requires all worker nodes to be in the same Layer 2 subnet.
• Security (NetworkPolicy): Flannel doesn’t support NetworkPolicy at all. Flannel has no code for filtering data packets. If we use Flannel, all Pods in the cluster can freely contact other Pods without limits.
• Best Use Cases: Local development clusters (like Kind, Minikube, or k3s), small non-production clusters that don’t need network security isolation, or worker nodes with very limited hardware specs.

2. Calico: The Production Industry Standard #

Calico (by Tigera) is the most widely adopted CNI for enterprise-scale production clusters. Calico is designed with a focus on high-level security and maximum routing performance.

Calico Architecture (BGP Direct Routing - Zero Encapsulation):
  [Pod A (10.244.1.5)] ──> [Host Route Table] ──> [Physical Router (BGP Route Advertisement)] ──> Destination Worker Node

• Routing Mechanism: Calico’s main advantage is its ability to do direct routing without encapsulation using the BGP (Border Gateway Protocol) protocol. Calico runs a BIRD daemon on every worker node to dynamically advertise Pod CIDR routes to data center physical routers. If the infrastructure doesn’t support BGP, Calico can be configured using IP-in-IP or VXLAN overlays.
• Security (NetworkPolicy): Calico has a very strong security module. It fully supports the standard Kubernetes NetworkPolicy spec and provides additional Calico NetworkPolicy objects supporting enterprise features (like GlobalNetworkPolicy, policy ordering, and IP set integration).
• Best Use Cases: Large-scale production clusters, on-premise bare-metal environments with physical network router integration, multi-tenant clusters needing strict micro-segmentation firewall isolation.

3. Cilium: The Next-Generation eBPF Pioneer #

Cilium is a revolutionary modern CNI leveraging the newest Linux kernel technology called eBPF (extended Berkeley Packet Filter). eBPF lets us run safe sandboxed programs directly inside the Linux kernel dynamically, without changing kernel source code or loading new kernel modules.

Cilium Architecture (eBPF Data Path - Bypass iptables & kube-proxy):
  [Pod A] ──> [eBPF Program in Kernel Space (Routing & Policy)] ──> [eBPF Bypass] ──> Physical Network Card

• Routing Mechanism: Cilium uses eBPF to filter and route packets directly at the kernel socket level. Cilium can fully replace the kube-proxy component (kube-proxy replacement). With eBPF, Cilium skips the long, slow iptables rule stack, redirecting packets instantly using very fast internal kernel lookup tables.
• Security (NetworkPolicy): Cilium supports whitelisting up to Layer 7 (application). We can write NetworkPolicies restricting API queries (e.g. the frontend Pod may only do HTTP GET queries to /public, and is blocked from POST to /admin).
• Observability (Hubble): Cilium integrates with Hubble, an outstanding network observability system. Hubble provides real-time graphical visualization of cluster traffic, interactive maps of inter-Pod connections, and logs rejected data packets with their reasons.
• Best Use Cases: Modern large-scale clusters (hundreds to thousands of nodes), latency-sensitive applications, strict cybersecurity audit needs, and clusters that want to run without iptables overhead.

CNI Technical Comparison Table #

Here’s a comprehensive evaluation table to help us choose the right CNI:

CNI Migration Procedures on Active Clusters #

[!CAUTION] Replacing a CNI plugin on a running production cluster (live CNI migration) is a high-risk operation triggering connection disruptions (downtime). This happens because there’s no hot-swap CNI method. When the old CNI is turned off and a new CNI is turned on, old Pods running with the old CNI routing configuration lose communication routes to new Pods.

If forced to do a CNI migration (e.g. migrating from Flannel to Cilium):

1. Plan a Maintenance Window: Make sure external client applications have been redirected to a backup cluster or use a maintenance page during the process.
2. Drain Nodes Gradually: Empty worker nodes one by one using the kubectl drain command so all Pods move to other nodes.
3. Clean Up Old CNI Configuration on Hosts: On every emptied worker node:
   • Delete the old CNI DaemonSet object from the API Server.
   • Delete the CNI configuration files in the host /etc/cni/net.d/ directory.
   • Delete the CNI binaries in /opt/cni/bin/.
   • Run the containerd / docker engine restart command on the host node.
4. Apply the New CNI Plugin: Deploy the new CNI manifests (or use Helm) to the cluster.
5. Uncordon Nodes: Reactivate worker nodes using kubectl uncordon.
6. Repeat for All Nodes: Alternate this process for every worker node until the whole cluster uniformly uses the new CNI.

CNI Implementation Anti-Patterns vs Solutions #

Let’s study some architectural mistakes (anti-patterns) that often happen when configuring CNIs, along with how to fix them.

Anti-Pattern 1: Ignoring Network Security by Choosing the Flannel CNI #

We deploy a multi-tenant production cluster using the Flannel CNI because its installation process is the easiest. Then we write dozens of complex NetworkPolicy manifests to separate traffic between developer teams.

The Bad Consequences #

The Kubernetes API Server accepts and stores our NetworkPolicy manifests without error messages. However, because the Flannel CNI doesn’t support packet filtering, all those NetworkPolicy rules are silently ignored. Developers think their databases are safely isolated, when in reality all Pods in the cluster remain free to access those database ports without restrictions.

Solution Code (Using a CNI That Enforces NetworkPolicies) #

We must install the Calico or Cilium CNI so all NetworkPolicy manifests we write are actually executed at the worker node host kernel level.

# SOLUTION: Example Cilium Helm installation snippet guaranteeing NetworkPolicy enforcement
# helm install cilium cilium/cilium \
#   --namespace kube-system \
#   --set networkPolicy.enabled=true \
#   --set networkPolicy.http=true # Enables eBPF Layer 7 HTTP filtering

Anti-Pattern 2: Using Calico BGP Mode on Cloud VPCs Without an Encapsulation Fallback #

We configure the Calico CNI with direct routing mode (BGP direct routing) on an EKS cluster in AWS. We turn off the IP-in-IP encapsulation feature to chase maximum data transfer performance.

Wrong Manifest Code (BGP Without Encapsulation on a Cloud VPC) #

# DON'T DO THIS ON CLOUD PROVIDERS: Packets are prone to being dropped
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 10.244.0.0/16
  ipipMode: Never # DON'T: AWS VPC routers drop cross-subnet Pod IP packets!
  natOutgoing: true

The Bad Consequences #

Internal cloud network routers (like AWS VPC) enforce a strict security rule called Source/Destination Check. Cloud routers drop every data packet crossing subnet boundaries whose sender IP isn’t officially registered in the EC2 virtual network card routing table. As a result, Pods on Worker Node A can’t reach Pods on Worker Node B in a different AWS subnet.

Solution Code (Using Cross-Subnet / Hybrid Mode) #

We must enable IP-in-IP encapsulation smartly using CrossSubnet mode. Data packets are sent directly without encapsulation if Pods are on worker nodes in the same VPC subnet, and automatically encapsulated only when packets must cross cloud VPC subnet boundaries.

# SOLUTION: Enable CrossSubnet mode for smooth cloud routing
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 10.244.0.0/16
  ipipMode: CrossSubnet # SAFE: Automatically encapsulates only when crossing cloud subnet boundaries
  natOutgoing: true

Anti-Pattern 3: Running the kube-proxy DaemonSet Alongside Cilium Strict eBPF Mode #

We deploy the Cilium CNI with full bypass configuration kubeProxyReplacement=strict to replace kube-proxy with eBPF, but leave the cluster’s built-in kube-proxy DaemonSet running.

The Bad Consequences #

Both components fight for network rule control in the worker node host kernel. kube-proxy continuously writes tens of thousands of iptables rules, while Cilium tries to bypass them using eBPF socket maps. This triggers kernel memory waste, route rule collisions, and erratic network I/O performance degradation.

Solution Code (Remove kube-proxy and Set Strict eBPF Replacement) #

We must cleanly delete the kube-proxy DaemonSet from the cluster (or disable it using a non-matching node selector), then install Cilium with full replacement parameters.

# SOLUTION: 1. Delete the kube-proxy DaemonSet from the cluster
kubectl delete daemonset kube-proxy -n kube-system

# 2. Install Cilium with STRICT kube-proxy replacement
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=api.k8s.mycompany.com \
  --set k8sServicePort=6443

• With the kubeProxyReplacement=strict parameter, Cilium takes over all ClusterIP and NodePort Service load balancing functions directly inside kernel space eBPF programs without touching iptables.

Summary #

• Kubernetes delegates networking to the CNI: The Kubelet calls the CNI binary in the /opt/cni/bin/ directory with ADD and DEL commands to configure Pod network namespaces, IPAM, and routing rules.
• Flannel is meant for simplicity: Quick to install, but has no NetworkPolicy support and medium performance due to VXLAN overhead.
• Calico is the production workhorse CNI: Provides high performance via native BGP routing and has a very strong NetworkPolicy enforcement module.
• Cilium pioneers the eBPF data path: Bypasses the iptables stack and kube-proxy directly in kernel space; supports Layer 7 HTTP filtering and Hubble UI observability.
• Use CrossSubnet mode in the cloud: Avoid pure native routing mode without encapsulation on cloud providers to prevent packet drops from cloud router Source/Destination Checks.
• CNI migrations trigger downtime: There’s no CNI hot-swap; plan gradual node drain processes and manually clean up host configuration remnants during maintenance windows.

← Previous: Load Balancing & kube-proxy   Next: Service Mesh →