Pod-to-Pod Communication #

The Kubernetes network model guarantees that every Pod has its own unique IP address and all Pods can communicate directly without Network Address Translation (NAT). However, for those of us responsible for keeping production infrastructure stable, merely knowing this “contract” isn’t enough. We must deeply understand the physical mechanisms of how data packets travel across virtual network cards, virtual kernel bridges, and physical network encapsulation tunnels.

Understanding the packet flow journey from one Pod to another is our main foundation for diagnosing connectivity issues (network troubleshooting), tuning data transfer performance, and understanding why abstraction objects like Service and DNS were created.

This article deeply covers the two main inter-Pod communication scenarios: communication within the same worker node (intra-node) and communication across worker nodes (inter-node), plus network testing techniques.

Scenario 1: Intra-Node Communication (Pods on the Same Node) #

When two Pods live on the same physical server or VM worker node, data packets never leave toward that node’s physical network card. The entire packet delivery process is fully handled inside the memory and Linux host kernel using a combination of veth pairs and a virtual bridge.

Let’s look at the internal intra-node communication architecture through the following diagram:

flowchart TD
    subgraph PodA["Pod A Namespace (10.244.1.2)"]
        eth0A["eth0 (Pod A)"]
    end

    subgraph PodB["Pod B Namespace (10.244.1.3)"]
        eth0B["eth0 (Pod B)"]
    end

    subgraph HostNamespace["Worker Node 1 Host"]
        cbr0["Virtual Bridge: cbr0 (Linux Bridge)"]
        vethA["veth-a (Host-end)"]
        vethB["veth-b (Host-end)"]
    end

    eth0A <--> vethA
    vethA <--> cbr0
    cbr0 <--> vethB
    vethB <--> eth0B

Key Components of Node Local Networking #

  1. veth pair (Virtual Ethernet Pair): We can imagine a veth pair as a two-way virtual network cable. When the Kubelet creates a new Pod, it creates a pair of veth interfaces. One end of this virtual cable is inserted into the Pod’s network namespace (named eth0 inside the Pod), while the other end stays in the host node’s namespace (given a random name like veth-a or veth-b).
  2. Linux Bridge (cbr0): This is a software-based virtual Ethernet switch device running inside the Linux host kernel. All veth cable ends in the host namespace (like veth-a and veth-b) plug into this bridge. The bridge acts as the intermediary connecting all those virtual network interfaces into one common local LAN segment.

The Data Packet Journey (Step-by-Step) #

Let’s trace the packet delivery flow when Pod A (10.244.1.2) sends an HTTP request to Pod B (10.244.1.3):

  1. Route Determination by Pod A: The kernel inside Pod A’s namespace checks the destination IP (10.244.1.3). Because that IP is in the same local subnet (e.g. 10.244.1.0/24), the kernel concludes the packet can be sent directly to the default gateway, which is Pod A’s eth0 interface.
  2. Crossing the Virtual Cable: The data packet leaves Pod A’s eth0 and instantly appears at the paired cable end in the host namespace, the veth-a interface.
  3. Bridge Decision: The packet arrives at the cbr0 virtual bridge. This bridge acts like a layer-2 physical switch. It reads the destination MAC address from the packet. If the bridge doesn’t yet know which interface holds that destination MAC, it broadcasts an ARP (Address Resolution Protocol) query to all connected ports: “Who owns the IP address 10.244.1.3?”.
  4. Response and Delivery: The veth-b interface (Pod B’s end) responds with its MAC address. The cbr0 bridge then updates its internal MAC address table and forwards the packet directly to the veth-b port.
  5. Reception by Pod B: The packet crosses the veth-b virtual cable and appears at Pod B’s eth0 interface. Pod B’s kernel receives the packet and forwards it to our application port.

All this communication happens at local memory speed without involving physical network interface card (NIC) latency, so latency is extremely low (under 0.1 ms).

Scenario 2: Inter-Node Communication (Pods Across Nodes) #

When Pod A on Node 1 wants to reach Pod C on Node 2, the communication flow becomes much more complex. The data packet must now leave Node 1’s physical server, cross the data center’s physical network switches or cloud VPC infrastructure, and land on Node 2.

To make this happen, we heavily depend on the CNI routing configuration. There are two main approaches CNIs commonly use to connect Pods across nodes: Overlay Networks (Encapsulation) and Native Routing (Direct Routing).

Approach A: Overlay Networks (VXLAN Encapsulation) #

The overlay network method creates a virtual logical network on top of the existing physical network. CNIs like Flannel (by default), or Calico/Cilium with VXLAN mode enabled, wrap the original Pod data packet inside a host node’s physical network UDP packet.

Let’s look at how packets are encapsulated crossing the physical network:

flowchart TD
    subgraph Node1["Worker Node 1 (10.0.0.10)"]
        PodA["Pod A (10.244.1.5)"] --> Netns1["veth-a"]
        Netns1 --> Bridge1["cbr0"]
        Bridge1 --> Tunnel1["VXLAN Tunnel Interface (flannel.1)"]
        Tunnel1 --> NIC1["Physical NIC (eth0)"]
    end

    subgraph PhysicalNet["Physical Network Infrastructure / Cloud VPC"]
        NIC1 -- "Encapsulated UDP Packet (Port 4789)" --> NIC2
    end

    subgraph Node2["Worker Node 2 (10.0.0.11)"]
        NIC2["Physical NIC (eth0)"] --> Tunnel2["VXLAN Tunnel Interface (flannel.1)"]
        Tunnel2 --> Bridge2["cbr0"]
        Bridge2 --> Netns2["veth-c"]
        Netns2 --> PodC["Pod C (10.244.2.3)"]
    end

The Packet Encapsulation Flow (Step-by-Step) #

  1. Local Routing Failure: Pod A (10.244.1.5) sends a packet to Pod C (10.244.2.3). Node 1’s kernel sees the destination IP is outside Node 1’s local subnet (10.244.1.0/24). The packet goes to the host’s default gateway interface.
  2. Capture by the CNI Tunnel: Node 1’s route table directs 10.244.2.0/24 subnet traffic to the CNI’s virtual tunnel interface (e.g. flannel.1 or cilium_vxlan).
  3. The Encapsulation (Wrapping) Process: The CNI virtual interface takes the original packet and wraps it inside a new UDP packet.
    • Inner Header: Shows the original sender 10.244.1.5 (Pod A) and original destination 10.244.2.3 (Pod C).
    • Outer Header: Shows the physical sender IP 10.0.0.10 (Node 1) and physical destination IP 10.0.0.11 (Node 2) with the standard VXLAN UDP destination port (port 4789).
  4. Travel on the Physical Network: Physical routers or cloud VPC switches only see a normal UDP packet from Node 1 to Node 2. They don’t need to know about Pod IPs. The packet is sent like ordinary server-to-server traffic.
  5. Decapsulation at the Destination Node: The packet lands on Node 2. Node 2’s kernel sees the packet is addressed to VXLAN port 4789, then forwards it to the local CNI driver. The CNI driver unwraps the UDP packet, takes the original packet inside, and throws it onto Node 2’s virtual bridge to be delivered directly to Pod C.

Overhead and MTU Optimization #

VXLAN encapsulation adds 50 bytes of overhead to every packet (IP + UDP + VXLAN headers). If our physical network interface has a standard packet size limit (MTU - Maximum Transmission Unit) of 1500 bytes, the MTU inside our Pods must be lowered to 1450 bytes. Otherwise, packets fragment at the host node level, triggering significant network throughput performance degradation.

Approach B: Native Routing (Direct Routing with BGP) #

In the native routing approach, we don’t use packet encapsulation. There’s no UDP wrapping overhead. Packets are sent raw as-is across the physical network. CNIs like Calico use the BGP (Border Gateway Protocol) protocol to turn each worker node into a dynamic router advertising its Pod CIDR routes to our data center’s physical network routers.

The Native Routing Flow (Step-by-Step) #

  1. Direct Sending: Pod A (10.244.1.5) sends a packet to Pod C (10.244.2.3).
  2. Host Route Table Check: Node 1’s kernel looks at the routing table configured by Calico: “To reach 10.244.2.0/24, send the packet directly through the node’s physical default gateway” (without wrapping the packet).
  3. Routing by the Physical Router: Our physical network router already received BGP advertisements from Node 2 earlier: “The 10.244.2.0/24 block sits behind host IP 10.0.0.11”. The physical router directly deflects that raw packet to Node 2.
  4. Direct Reception: Node 2 receives the raw packet and directly forwards it to Pod C via its local bridge.

Advantages & Limitations #

  • Advantages: Maximum data transmission performance (equivalent to physical bare-metal networking) and minimal latency because there’s no CPU load for encapsulation/decapsulation.
  • Limitations: Our data center’s physical routers must be configured to support BGP integration with worker nodes. On public cloud providers (like AWS VPC), cloud routers often drop packets whose sender IP addresses aren’t registered in the VPC routing table (the Source/Destination Check problem). Therefore, in the cloud, we often use mixed modes (cross-subnet or hybrid) where encapsulation is only enabled when packets must cross cloud router subnet boundaries.

Inter-Node Network Traffic Encryption #

By default, all data packets flowing between worker nodes travel in plain text. If our cluster is on a public cloud, there’s a risk our sensitive traffic (like transaction data, credentials, or user personal data) can be spied on by parties with hypervisor-level cloud access.

To secure that data, modern CNIs like Cilium and Calico provide transparent encryption options at the cluster level using WireGuard or IPsec.

Transparent Network Layer Encryption (CNI WireGuard):
  [Pod A (Plain Data)] 
         │
         ▼
  [Node 1 Kernel (Cilium WireGuard Interface)] ──> Packet Encryption (AES/ChaCha20)
         │
         ▼ (Encrypted Tunnel via the Physical Network)
  [Node 2 Kernel (Cilium WireGuard Interface)] ──> Packet Decryption
         │
         ▼
  [Pod C (Plain Data)]

By enabling this feature, the Linux host kernel automatically encrypts all inter-Pod data packets before they leave the worker node using high-speed modern cryptographic algorithms (like ChaCha20 for WireGuard), and decrypts them when they land at the destination node. Our applications need zero code modifications because security is guaranteed directly at the network infrastructure level.

Inter-Pod Communication Anti-Patterns vs Solutions #

Let’s study some fatal mistakes we often find when managing inter-Pod communication, along with their manifest code comparisons.

Anti-Pattern 1: Hardcoding Pod IP Addresses to Reach Other Services #

We put the database Pod’s IP address (10.244.1.7) directly into our backend Pod’s environment variable configuration.

Wrong Manifest Code (Hardcoded Ephemeral IP) #

# DON'T DO THIS IN PRODUCTION: Pod IPs change easily
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-app-bad
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: app-container
          image: my-app:v1.0.0
          env:
            - name: DATABASE_URL
              value: "postgresql://postgres:[email protected]:5432/db" # DON'T: The database Pod IP is unstable!

The Bad Consequences #

If the database Pod crashes in the middle of the night, Kubernetes restarts a new healthy database Pod on another worker node. However, this new database Pod gets a new IP address (e.g. 10.244.2.12). As a result, our backend Pods completely lose their connection and keep triggering database connection errors because they’re still trying to reach the old IP 10.244.1.7.

Solution Code (Using the Service DNS Abstraction) #

We must deploy a Service object in front of our database Pods. The Service provides a stable virtual IP (ClusterIP) and an internal DNS name that never changes as long as the Service object is active, no matter how many times the Pods behind it restart or change IP addresses.

# SOLUTION: Always use the internal Service DNS name
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-app-good
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: app-container
          image: my-app:v1.0.0
          env:
            - name: DATABASE_URL
              value: "postgresql://postgres:[email protected]:5432/db" # Stable credentials via the Service DNS

Anti-Pattern 2: Ignoring MTU Alignment on Cloud Overlay CNIs #

We deploy a self-managed Kubernetes cluster on AWS cloud using the Calico CNI with VXLAN encapsulation enabled, but keep the CNI MTU setting at the host physical ethernet default of 1500 bytes.

The Bad Consequences #

Every time our application sends a full-size data packet (1500 bytes), the CNI driver adds a 50-byte VXLAN encapsulation header, ballooning the packet size to 1550 bytes. Because the cloud VPC network caps physical packets at 1500 bytes, the host node is forced to split one packet into two (packet fragmentation) at the host kernel level. This drastically wastes node CPU on fragmentation/reassembly and degrades network throughput performance by up to 30%.

Solution Code (Configuring MTU Precisely on the CNI) #

We must align the MTU size in the CNI plugin configuration to leave room for the encapsulation header. For VXLAN, reduce the MTU by 50 bytes from the host physical MTU (1500 - 50 = 1450).

# SOLUTION: Example Calico CNI ConfigMap configuration snippet
apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Optimized MTU configuration for VXLAN overlay
  veth_mtu: "1450" # Ensures in-Pod packets max at 1450, so the total physical packet after encapsulation fits exactly 1500

Practical Guide to Testing & Debugging Pod Networking #

When inter-Pod communication issues occur in the cluster, we can do systematic investigation directly from the terminal using network debugging tools.

1. Create an Isolated Debug Pod (netshoot) #

Often our production application images are deliberately designed to be very minimal (distroless) for security, so they lack utilities like ping, curl, or nslookup. We can launch a special debug container named netshoot that comes with all network analysis tools:

kubectl run network-debugger --rm -i --tty --image nicolaka/netshoot -- /bin/bash

2. Test Layer 3 Connectivity (IP Routing) #

From inside the debug Pod, ping the destination Pod IP directly to verify the CNI inter-node routing functionality:

# Ping the destination Pod IP on another worker node
ping -c 4 10.244.2.3

3. Trace Packet Hops (Tracepath) #

If ping fails, use tracepath or traceroute to see at which router hop our data packets get dropped:

tracepath -n 10.244.2.3

4. Monitor Physical Worker Node Packets #

If we have SSH access to the worker node host, we can use tcpdump on the node’s physical network card to verify whether encapsulated VXLAN packets (port 4789) successfully leave the sending node:

# Monitor VXLAN traffic on the worker node host
sudo tcpdump -i eth0 -n udp port 4789

Summary #

  • Intra-node communication runs fast in memory: Pods on the same node communicate through a local virtual ethernet bridge (cbr0) using veth pairs without touching the physical network.
  • Overlay networks use VXLAN encapsulation: Original packets are wrapped in host UDP port 4789 packets for cross-node delivery; requires MTU adjustment to 1450 bytes to avoid fragmentation.
  • Native BGP routing eliminates overhead: Connects Pods directly through the physical network using real IP routes; requires tight integration with physical/cloud VPC router configurations.
  • Transparent WireGuard encryption: Modern CNIs support automatic kernel-level encryption for inter-node traffic to protect sensitive data from physical network sniffing.
  • Always use Service DNS: Pod IPs are ephemeral and prone to change when Pods restart. Always use stable Service DNS names for inter-service communication.
  • Use the netshoot debug container: Use kubectl run with a dedicated debugger image to interactively diagnose routing, DNS, and port firewall issues.

← Previous: Kubernetes Network Model   Next: Service →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact