Load Balancing & kube-proxy #

In the Kubernetes ecosystem, the Service object provides a virtual IP address (ClusterIP) and a stable internal DNS name for a group of dynamic Pods. When an application sends an HTTP request to that Service’s virtual IP address, there’s a behind-the-scenes mechanism tasked with distributing the traffic evenly to one of the backend Pods.

The main component responsible for managing this internal load balancing is kube-proxy. kube-proxy is a network agent running as a DaemonSet on every worker node in our cluster.

One important concept we must understand from the start: kube-proxy doesn’t process traffic directly. kube-proxy isn’t in the main data path. kube-proxy acts as a network rule manager (control agent) tasked with configuring the local Linux kernel so that kernel itself performs instant packet redirection at the host VM kernel level.

This article covers the kube-proxy architecture, how load balancing works in iptables and IPVS modes, their performance differences, and external traffic customization techniques.

How kube-proxy Works: Watching the API Server #

kube-proxy works by continuously watching the Kubernetes API Server. It listens for every creation, change, or deletion of Service, Endpoints, and EndpointSlice objects in the cluster.

When a new Service is created:

  1. The API Server allocates a virtual IP address (ClusterIP, e.g. 10.96.45.12) for that Service.
  2. The Kubernetes Controller Manager records all healthy Pod IP addresses matching the Service’s selector into the Endpoints list.
  3. kube-proxy on every worker node catches that event, then writes new network rules (routing rules) into the local Linux kernel firewall on each of its worker nodes.

By letting the Linux host kernel redirect data packets, Kubernetes avoids performance bottlenecks. Packets get redirected at kernel hardware speed without context switching to user-space application processes.

Deep Dive into iptables Mode (Default Mode) #

By default, kube-proxy is configured using iptables mode. iptables is a built-in Linux kernel firewall administration utility tasked with cutting, modifying, and redirecting data packets based on structured rules.

When using iptables mode, kube-proxy creates a series of special rule chains for every Service in the cluster.

Let’s look at how these iptables rule chains are arranged by kube-proxy:

flowchart TD
    Packets["Incoming Packet (Target: 10.96.45.12:80)"] --> KService["Chain: KUBE-SERVICES"]
    KService -- "Match ClusterIP" --> KServiceChain["Chain: KUBE-SVC-API-SERVICE"]
    
    KServiceChain -- "33% Probability" --> KEPodA["Chain: KUBE-SEP-POD-A"]
    KServiceChain -- "50% (Remainder)" --> KEPodB["Chain: KUBE-SEP-POD-B"]
    KServiceChain -- "Fallback (Remainder)" --> KEPodC["Chain: KUBE-SEP-POD-C"]
    
    KEPodA --> DNATA["DNAT to Pod A (10.244.1.5:8080)"]
    KEPodB --> DNATB["DNAT to Pod B (10.244.2.3:8080)"]
    KEPodC --> DNATC["DNAT to Pod C (10.244.1.9:8080)"]

1. The iptables Chain Flow (Step-by-Step) #

Let’s break down the rule chain above when a client calls the api-service Service (10.96.45.12:80) with 3 backend Pod replicas:

  1. KUBE-SERVICES: The data packet is first caught by the main KUBE-SERVICES chain. This chain detects: “Does this packet’s destination IP match our Service’s ClusterIP?”. If it matches (destination 10.96.45.12), jump to that Service’s dedicated chain: KUBE-SVC-API-SERVICE.
  2. KUBE-SVC-API-SERVICE (Random Probability): Because the Service has 3 backend Pods, this chain uses the kernel statistic module to randomly distribute traffic based on probability values:
    • Line 1: With 0.33 (33%) probability, jump to the KUBE-SEP-POD-A chain (Pod A).
    • Line 2: With 0.50 (50% of the remaining 66% traffic) probability, jump to the KUBE-SEP-POD-B chain (Pod B). This value equals 33% of total overall traffic.
    • Line 3: If both lines above are passed (remaining traffic), jump directly to the KUBE-SEP-POD-C chain (Pod C).
  3. KUBE-SEP-POD-A (Destination NAT): This endpoint chain executes the DNAT (Destination Network Address Translation) command to change the packet’s destination IP from the virtual Service IP 10.96.45.12:80 to the real target Pod IP, 10.244.1.5:8080.

Key iptables Mode Limitations #

Although iptables mode is very mature and stable, it has serious architectural weaknesses when run on large-scale clusters:

  • Sequential O(n) Lookup: iptables evaluates rules linearly from top to bottom. If our cluster has 10,000 Services with 5 Endpoints each, the Linux kernel must evaluate up to tens of thousands of rules for every single incoming packet. This triggers cluster network latency spikes and worker node CPU usage.
  • Non-Atomic Updates: When one backend Pod dies or changes IP, kube-proxy must flush the worker node’s entire iptables table and rewrite all rules from scratch. This non-atomic process can trigger momentary packet drops (latency spikes) on very busy clusters.

Deep Dive into IPVS Mode (IP Virtual Server) #

To overcome the iptables mode performance limitations on large clusters, Kubernetes provides an alternative mode called IPVS (IP Virtual Server). IPVS is part of Linux Virtual Server (LVS), embedded in the Linux kernel for a long time and specifically designed for high-performance load balancing.

IPVS uses a Hash Table data structure to store Service routing rules instead of a linear sequential list.

Let’s look at the lookup complexity comparison between iptables and IPVS modes through the following diagram:

flowchart TD
    subgraph iptablesMode["iptables Mode (Linear Lookup - O(n) Complexity)"]
        Packets1["Incoming Packet"] --> Rule1{"Rule 1?"}
        Rule1 -- "No" --> Rule2{"Rule 2?"}
        Rule2 -- "No" --> RuleN{"Rule N?"}
        RuleN -- "Yes" --> Dest1["Redirect to Pod"]
    end

    subgraph ipvsMode["IPVS Mode (Hash Table Lookup - O(1) Complexity)"]
        Packets2["Incoming Packet"] --> HashKey["Hash Key: ClusterIP:Port"]
        HashKey --> HashTable{{"Hash Table Lookup"}}
        HashTable --> Dest2["Redirect to Pod (Instant)"]
    end

IPVS Technical Advantages #

  • Constant O(1) Complexity: No matter how many Services we have in the cluster (whether 100 or 100,000 Services), IPVS only needs one instant hash lookup to find the destination Pod. Network query latency stays constant and very low.
  • Atomic Updates: IPVS routing table updates are done incrementally (only changing the rows of changed Pod IPs) and are atomic. No whole-table cleanup, minimizing packet drop risk during rolling updates.
  • Load Balancing Algorithm Choices: Unlike iptables which only supports random probabilistic selection, IPVS provides various advanced load balancing algorithms:
    • rr (Round-Robin): Divides traffic alternately in sequence.
    • lc (Least Connection): Sends new traffic to the Pod with the fewest active connections (highly recommended for databases/long-lived connections).
    • sh (Source Hashing): Maps traffic based on client IP hash (for sticky sessions).
    • dh (Destination Hashing): Maps based on destination IP.

How to Enable IPVS Mode #

We must update the kube-proxy ConfigMap configuration file in the kube-system namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system
data:
  config.conf: |-
    mode: "ipvs" # Enables IPVS mode
    ipvs:
      scheduler: "lc" # Uses the Least Connection algorithm

[!WARNING] Before enabling IPVS mode on a self-managed cluster, we must ensure the Linux kernel modules IPVS needs have been automatically loaded on our worker node host OSes. Those modules are: ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, and nf_conntrack.

External Traffic Policy #

For Services receiving traffic from outside the cluster (NodePort or LoadBalancer types), Kubernetes provides the .spec.externalTrafficPolicy property to regulate how that external traffic is processed across the cluster’s worker nodes.

spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster # The default choice
  # or:
  # externalTrafficPolicy: Local

1. externalTrafficPolicy: Cluster (Default) #

When an external data packet lands on Node A, but the Kubernetes scheduler placed our application Pod on Node B:

  • kube-proxy on Node A performs Source NAT (SNAT) to forward the packet across the internal cluster network to Node B.
  • Advantages: Traffic distribution is very even to all application Pods in the cluster.
  • Disadvantages: We lose the original external client IP because it gets overwritten by Node A’s internal IP during SNAT. Additionally, there’s extra latency because the packet must hop between worker nodes (network hop).

2. externalTrafficPolicy: Local #

When an external data packet lands on Node A:

  • kube-proxy on Node A only forwards the packet to Pods running inside Node A.

  • If our application Pod isn’t on Node A, the packet gets dropped.

  • Advantages: The original external client IP is fully preserved (Source IP preservation) because there’s no cross-node SNAT process. There’s also no additional cross-node network hops.

  • Disadvantages: Traffic distribution can become uneven if our Pod replica counts per worker node are unbalanced.

  • Note: For cloud LoadBalancer Services, the cloud provider automatically health-checks the special NodePort on every worker node. Worker nodes without active Pods fail the health check, so the cloud load balancer won’t send traffic to those nodes.

Cluster Load Balancing Anti-Patterns vs Solutions #

Let’s study some fatal mistakes related to kube-proxy and load balancing configuration, along with their code comparisons.

Anti-Pattern 1: Leaving iptables Mode Active on a Large-Scale Cluster #

We operate a production Kubernetes cluster hosting thousands of microservices with tens of thousands of active Service objects, but we leave kube-proxy running in the default iptables mode.

Wrong Manifest Code (Built-in iptables Mode) #

# DON'T USE THIS ON LARGE CLUSTERS: I/O performance will degrade
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system
data:
  config.conf: |-
    mode: "iptables" # Slow linear O(n) traversal for thousands of Services

The Bad Consequences #

Every time an application Pod sends a request to another Service, the worker node kernel needs significant CPU time to sequentially traverse tens of thousands of iptables rule lines. Internal cluster HTTP request latency balloons, and worker node CPU utilization gets drained just processing network firewall rules.

Solution Code (Migrating to IPVS Mode with the Least Connection Algorithm) #

# SOLUTION: Enable IPVS for constant O(1) lookup performance
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system
data:
  config.conf: |-
    mode: "ipvs" # Switch to hash table routing
    ipvs:
      scheduler: "lc" # Uses Least Connection for database workload optimization

Anti-Pattern 2: Using externalTrafficPolicy: Local Without Even Pod Distribution #

We enable the externalTrafficPolicy: Local property on a LoadBalancer Service to preserve the original client IP, but let our Pods pile up on only one worker node due to the lack of affinity rules.

Wrong Manifest Code (Local Service Without Topology Constraints) #

# DON'T DO THIS: Load Balancer traffic won't be balanced
apiVersion: v1
kind: Service
metadata:
  name: web-service-local
  namespace: production
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local # Packets are only processed on nodes with Pods
  selector:
    app: web-server
  ports:
    - port: 80
      targetPort: 80

The Bad Consequences #

If the cluster has 3 worker nodes, and all 3 of our application Pod replicas are scheduled by Kubernetes on Node 1 only, the cloud load balancer only sends traffic to Node 1. Nodes 2 and 3 receive no traffic at all. Node 1 suffers CPU overload while Node 2 and Node 3 resources are wasted.

Solution Code (Applying Topology Spread Constraints on Pods) #

We must add topologySpreadConstraints or podAntiAffinity configuration to our Pod Deployment manifests to guarantee our Pod replicas spread evenly across all cluster worker nodes.

# SOLUTION: Forcing the scheduler to spread Pods evenly across all worker nodes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-server
  template:
    metadata:
      labels:
        app: web-server
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule # Forces the scheduler not to pile Pods on one node
          labelSelector:
            matchLabels:
              app: web-server
      containers:
        - name: web-container
          image: nginx:alpine

Practical kube-proxy & IPVS Debugging Guide #

If Service traffic isn’t being distributed correctly, we can do internal verification on the worker node.

1. Check the kube-proxy DaemonSet Logs #

Get the list of kube-proxy Pod names running in the kube-system namespace:

kubectl get pods -n kube-system -l k8s-app=kube-proxy

Review one kube-proxy Pod’s logs to ensure it successfully loaded the network rules without errors:

kubectl logs -n kube-system pod/kube-proxy-abc123

Make sure there’s no error message like "Can't use ipvs mode, inserting iptables rules instead", which indicates the IPVS kernel modules on the worker node host haven’t been loaded.

2. Check the IPVS Table Directly on the Worker Node #

If we have SSH access to the worker node, we can install the ipvsadm utility to see the actual IPVS hash lookup table contents:

# Installing the IPVS administration utility on Debian/Ubuntu
sudo apt-get install -y ipvsadm

# Viewing the list of active IPVS load balancing rules
sudo ipvsadm -ln

The output displays the instant mapping of the virtual Service IP (ClusterIP) to the real target Pod IPs:

Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.45.12:80 lc
  -> 10.244.1.5:8080              Masq    1      0          0         # Pod A
  -> 10.244.2.3:8080              Masq    1      0          0         # Pod B
  -> 10.244.1.9:8080              Masq    1      0          0         # Pod C

Review whether the RemoteAddress column matches the list of healthy Pod IP addresses.

Summary #

  • kube-proxy manages rules, not packets: kube-proxy acts as a control agent writing routing rules on worker node hosts, while the Linux kernel redirects the data.
  • iptables uses linear O(n) lookup: Very mature, but performance degrades on large-scale clusters because it evaluates network rules sequentially.
  • IPVS uses O(1) Hash Tables: Highly recommended for large-scale production clusters because route lookup performance is constant and updates are atomic.
  • IPVS supports the Least Connection (lc) algorithm: Sends new traffic to the Pod with the fewest active connections; ideal for heavy database workloads.
  • externalTrafficPolicy: Local preserves the original Client IP: Avoids cross-worker-node Source NAT (SNAT) processes so applications can verify the real external client IP.
  • Spread Pods evenly when using the Local policy: Always use topologySpreadConstraints on Pod Deployments to prevent cross-worker-node load imbalance.
  • Verify IPVS with ipvsadm -ln: Use the IPVS administration utility directly on worker nodes to check smooth virtual Service IP to real container IP mappings.

← Previous: Network Policy   Next: CNI Plugin →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact