Network Troubleshooting #

Network problems inside a Kubernetes cluster are often one of the most complicated challenges faced by platform engineers and developers. This complexity happens because communication in Kubernetes involves many stacked abstraction layers, from virtual ethernet (veth) pairs at the container level, local bridges on the host, overlay network encapsulation by the CNI plugin (like VXLAN or Geneve), iptables/IPVS packet filtering rules by kube-proxy, domain name resolution by CoreDNS, L4 micro-segmentation security policies through NetworkPolicy, and L7 routing through the Ingress controller or Gateway API.

When connectivity disruptions occur, the symptoms are often vague, like connection timeouts, slow DNS resolution, or 502 Bad Gateway errors. Without a systematic tracing methodology, we get trapped in a guessing game cycle — like force-restarting application Pods or worker nodes randomly — which actually makes isolating the root cause harder. This article presents a step-by-step approach, essential diagnostic instruments, and analysis of common network failure scenarios in production environments to help us diagnose and resolve network problems effectively.

Methodology: Isolating Layers from the Outside In #

To diagnose network problems without losing direction, we must apply a systematic approach isolating each communication layer from the simplest (local connectivity) to the most complex (L7 routing and encryption). We start the investigation from individual Pod status, move to basic IP connectivity, then rise to DNS name resolution, load balancing rules, and finally traffic authorization policies.

This structured diagnostic flow helps us quickly eliminate possible problem causes, so we don’t waste time checking complicated Ingress configuration when the cluster’s CoreDNS is actually malfunctioning.

Let’s study the diagnostic decision tree flow below to understand the tracing steps from broad to specific conditions:

flowchart TD
    A["Start Network Investigation"] --> B{"1. Is the Pod Running & Ready?"}
    B -- "No" --> C["Resolve the Pod Lifecycle (Check Pending/CrashLoopBackOff)"]
    B -- "Yes" --> D{"2. Can it ping another Pod on the same Node?"}
    
    D -- "No" --> E["Check the CNI DaemonSet & Node Local veth Interfaces"]
    D -- "Yes" --> F{"3. Can it ping a Pod on a Different Node?"}
    
    F -- "No" --> G["Check Overlay Routing (VXLAN/Geneve) & Node Network Security"]
    F -- "Yes" --> H{"4. Does nslookup kubernetes.default succeed?"}
    
    H -- "No" --> I["Check CoreDNS Logs, Service, & NetworkPolicy Port 53"]
    H -- "Yes" --> J{"5. Does accessing the Service via ClusterIP IP succeed?"}
    
    J -- "No" --> K["Check kube-proxy Status & Node iptables/IPVS Rules"]
    J -- "Yes" --> L{"6. Does accessing the Service via DNS Name succeed?"}
    
    L -- "No" --> M["Check the ndots value & search domains in /etc/resolv.conf"]
    L -- "Yes" --> N{"7. Are the Service Endpoints filled with Healthy Pods?"}
    
    N -- "No" --> O["Check the Service Selector, Pod Labels, & Readiness Probe Status"]
    N -- "Yes" --> P{"8. Does Ingress / Gateway API routing succeed?"}
    
    P -- "No" --> Q["Check the Ingress Controller, Service TargetPort, & TLS Certificate"]
    P -- "Yes" --> R["The Network System Works Normally"]

Debug Toolkit: Essential Diagnostic Instruments #

To run the methodology above, we need the right instruments. In secure production Kubernetes environments, application containers are usually built from minimal images (distroless or alpine-slim) without basic network utilities like curl, ping, tcpdump, or nslookup to keep size efficiency and minimize attack surface. We can use several techniques below to inject debug tools into the cluster without damaging application integrity.

1. Debugging with Ephemeral Containers #

An ephemeral container is a special container type we can temporarily run inside an existing Pod to do administrative tasks like debugging. This container shares the same network, process, and volume namespaces as the main application container, so we can monitor network activity directly from within the Pod’s context without a restart.

We can use a comprehensive testing image like nicolaka/netshoot (containing complete tools like tcpdump, tshark, nmap, dig, curl, iperf, and bind-tools) with the following command:

# Run a debug ephemeral container inside the problematic application Pod
kubectl debug -it order-processor-85df9b7754-abcde \
  --image=nicolaka/netshoot \
  --target=order-processor-container

The --target parameter is very important because it lets the debug container share the process namespace with the target application container, so we can run commands like netstat or inspect /proc to see active socket connections made by the main application process.

2. Running a Standalone Debug Pod (Temporary Pod) #

If we just want to test global cluster DNS resolution, outbound connectivity, or scan specific service ports from inside the cluster network, we can create a standalone debug Pod that gets deleted right after our interactive session ends.

# Run a temporary interactive Pod in the default namespace
kubectl run netshoot-temp --image=nicolaka/netshoot -it --rm --restart=Never -- sh

# Run a debug Pod in a specific namespace (e.g. production)
kubectl run netshoot-temp -n production --image=nicolaka/netshoot -it --rm --restart=Never -- sh

The --rm flag ensures Kubernetes automatically cleans up this Pod object from etcd when we type the exit command in the terminal.

3. Checking Network Status from the Control Plane CLI #

Before jumping into packet analysis, we must verify the correctness of the network configuration data stored by the Kubernetes API Server using these commands:

# 1. Check Pod IP allocations and the Node where the Pod runs
kubectl get pods -o wide -n production

# 2. Check whether Service Endpoints are dynamically filled
kubectl get endpoints order-service -n production
kubectl get endpointslices -l kubernetes.io/service-name=order-service -n production

# 3. Describe the Service port configuration in detail
kubectl describe service order-service -n production

Scenario 1: Pods Fail to Communicate Across Nodes (CNI Connectivity) #

One of the most common problems after a new cluster installation or adding new worker nodes is Pods on Node A being unable to send data packets to Pods on Node B, even though inter-Pod communication inside Node A runs smoothly. The main symptom of this scenario is connections that immediately time out without a refusal response (connection refused).

Packet Encapsulation Problems and the Node Protocol Path #

When Pod A sends a packet to Pod B on a different node, the CNI plugin is responsible for encapsulating that packet (usually using the VXLAN protocol on UDP port 4789 or Geneve on UDP port 6081) or routing it directly using BGP. If the physical network outside Kubernetes (like a cloud provider VPC firewall or on-premise switches) blocks those UDP ports, or if there’s a PodCIDR subnet allocation conflict, the packet encapsulation fails to decapsulate at the destination node.

Cross-Node Connectivity Diagnostic Comparison #

Let’s look at the difference between the wrong handling approach (anti-pattern) and the structured handling approach (solution):

# ANTI-PATTERN: Force-restarting the node without diagnosing
# This doesn't solve the problem if the cause is a closed VPC firewall port.
sudo reboot  # ✗ Erases important logs and unnecessarily disrupts other workloads

# CORRECT: Verify CIDR allocation, CNI status, and overlay port connectivity
# Step 1: Check whether Pod IP (PodCIDR) allocations conflict across nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

# Step 2: Run tcpdump on the destination worker node to detect incoming VXLAN packets
# (Run on Node B's terminal / the destination node)
sudo tcpdump -i any port 4789 -n -vv

# Step 3: Test UDP port 4789 connectivity from Node A to Node B using nc
nc -vzu <IP-Node-B> 4789

If the nc command shows a stuck (timeout) connection status, we must open UDP port 4789 in our infrastructure firewall security system (like Security Groups on AWS/GCP or local ufw/firewalld firewall rules).

Scenario 2: DNS Resolution Failures (CoreDNS & NetworkPolicy) #

Our application suddenly shows errors like dial tcp: lookup inventory-db on 10.96.0.10:53: no such host or Temporary failure in name resolution. This problem is fatal because the application can’t find the database or downstream microservice location even though the target Pod is active and healthy.

How Name Resolution Works in Kubernetes #

The Kubelet automatically configures the /etc/resolv.conf file in every container, pointing the nameserver address to the CoreDNS Service’s ClusterIP (usually ending in .10 in the Service subnet). This file also defines the search domain parameters so applications can call other services using just short names:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

If a CoreDNS problem occurs, DNS resolution stops entirely. This can be caused by the CoreDNS Pod suffering memory exhaustion (OOMKilled), too-high query loads from inefficient ndots configuration, or a new NetworkPolicy blocking outgoing Pod traffic to port 53 UDP/TCP in the kube-system namespace.

DNS Issue Handling Comparison #

Here’s a code comparison visualization between a bad bypass approach and proper root cause diagnosis:

# ANTI-PATTERN: Writing external DNS directly in the Pod manifest to bypass CoreDNS
# ✗ This breaks cluster portability principles and cuts access to internal cluster Services
apiVersion: v1
kind: Pod
metadata:
  name: web-app-bad
spec:
  containers:
  - name: app
    image: my-app:v1
  dnsConfig:
    nameservers:
      - 8.8.8.8  # Bypassing CoreDNS - The app can no longer access database-service.local!
---
# CORRECT: Create a NetworkPolicy explicitly allowing DNS queries to CoreDNS
# ✓ This maintains zero-trust security without sacrificing internal DNS functionality
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: production
spec:
  podSelector: {}  # Applies to all Pods in this namespace
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns  # Points to the CoreDNS Pod
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

To diagnose whether CoreDNS works well, we can run an interactive DNS resolution test command from a debug Pod:

# Query DNS to the internal Service using the full FQDN
dig @10.96.0.10 kubernetes.default.svc.cluster.local

# Check CoreDNS logs to detect forwarding errors or query loops
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

Scenario 3: ClusterIP Services Are Inaccessible (Kube-Proxy & Endpoints) #

We have a ClusterIP-type Service configured to expose a backend application. However, when the frontend tries sending HTTP requests to that ClusterIP, the connection always ends in a connection refused or timeout status. Interestingly, if we try contacting the backend Pod IP directly, the connection runs smoothly.

Kube-proxy and the Virtual IP Packet Forwarding Mechanism #

A ClusterIP isn’t a real physical network interface. The ClusterIP is just a virtual IP (VIP) rule registered in the Node’s kernel memory by the kube-proxy agent. Kube-proxy operates in two main modes:

  1. iptables mode: Kube-proxy writes long iptables chains to intercept packets destined for the ClusterIP, then does address translation (DNAT) randomly to one healthy backend Pod IP.
  2. IPVS mode: Kube-proxy uses the IPVS hash table at the Linux kernel level, which is much more efficient for large-scale clusters with thousands of services.

If we can’t access the Service via ClusterIP, the likely causes are:

  • Empty Endpoints: No backend Pods pass the Readiness Probe check, so the Service has no routing target.
  • Wrong Selector: Labels defined in the Service selector don’t match labels attached to the application Pods.
  • Stuck Kube-proxy: The kube-proxy agent on the node where the client runs fails to sync iptables/IPVS rules from the API Server.

Service Diagnostic Comparison #

Let’s compare a wrong fix action with proper routing rule verification steps:

# ANTI-PATTERN: Writing Pod IPs statically in the app to bypass the Service
# ✗ This breaks cluster scalability because Pod IPs are ephemeral
curl http://10.244.2.35:8080  # Contacting a Pod IP directly with hardcode

# CORRECT: Verify label selector matching and check iptables rules on the worker Node
# Step 1: Check whether active endpoints are bound to our Service
kubectl get endpoints order-service

# Step 2: If endpoints are empty, compare the Service selector with actual Pod labels
kubectl get service order-service -o jsonpath='{.spec.selector}'
kubectl get pods -l app=order-processor --show-labels

# Step 3: Check the iptables NAT rules created by kube-proxy inside the worker node
# (Run with root access on the worker Node where the client Pod runs)
sudo iptables-save -t nat | grep order-service

Correct iptables rules show chain entries like KUBE-SVC-XXXX performing probability routing (statistic mode random) to backend Pod IPs registered in the endpoints.

Scenario 4: Ingress Returns 502 / 503 Errors (Upstream Mismatch) #

External users trying to access our application through a public domain get a 502 Bad Gateway or 503 Service Unavailable error page coming from the Ingress Controller (like ingress-nginx).

HTTP/1.1 502 Bad Gateway
Server: nginx/1.25.3
Content-Type: text/html
Content-Length: 150

Distinguishing the Failure Location: Ingress Controller to Service, or Service to Pod #

A 502/503 error indicates the Ingress Controller successfully received the request from the internet, but failed to forward the packet to the backend application (upstream). This failure location can be isolated by checking the port relationships between the Ingress, Service, and Pod:

[Client] ---> (Port 443) Ingress ---> Service (Port 80) ---> Pod (TargetPort 8080)
                                      ^                      ^
                                  Wrong Port?            Crash/Not Ready?

Often, developers write the service.port configuration wrong in the Ingress manifest, pointing to the container port directly (targetPort) instead of the Service’s logical port (port), or the application inside the container dies suddenly so no socket is listening on the target port.

Ingress Routing Configuration Comparison #

Let’s study the routing manifest fix example below:

# ANTI-PATTERN: Connecting the Ingress directly to the application container's targetPort
# ✗ This is wrong because the Ingress must point to the logical port defined on the Service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-ingress-bad
spec:
  rules:
  - host: orders.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 8080  # ✗ WRONG: This is the container targetPort, not the Service port!
---
# CORRECT: Consistently align the Ingress port with the Service's public port
# ✓ The Ingress points to Service port 80, which the Service then forwards to the Pod targetPort
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-ingress-good
spec:
  rules:
  - host: orders.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 80  # ✓ CORRECT: This port matches Service spec.ports[0].port

To verify this connectivity directly, we can use the port-forwarding technique to bypass all Ingress and Service layers:

# Forward local port 9000 traffic directly to container port 8080 on the Pod
kubectl port-forward pod/order-processor-85df9b7754-abcde 9000:8080

# Test the app from our local terminal
curl -I http://localhost:9000/healthz

If the test above succeeds but the Ingress still returns a 502 error, we must check the Ingress Controller log files to see the upstream connection failure details:

kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50

Scenario 5: Mysterious Packet Loss (Detecting NetworkPolicy Drops) #

We just implemented a zero-trust security framework by installing a default-deny NetworkPolicy to isolate the production database environment. However, after that policy was installed, our backend microservice lost the ability to save data to the PostgreSQL database. There are no clear network error logs on the backend side besides a database connection timeout message.

Monitoring Packet Drops at the Kernel Level #

NetworkPolicies are implemented by the CNI plugin using Linux kernel modules (like iptables/ipset matching rules or eBPF programs loaded into container network interfaces). When a data packet is blocked by a NetworkPolicy, the OS silently drops that packet without sending an ICMP refusal reply to the sender. As a result, the sender application only detects a timeout status.

If we use a modern eBPF-based CNI like Cilium, we have the outstanding ability to monitor these packet drops in real-time directly at the kernel level using the Cilium monitor utility.

Network Drop Analysis Comparison #

Let’s compare a guesswork approach with advanced packet drop monitoring techniques:

# ANTI-PATTERN: Deleting the NetworkPolicy in production for testing
# ✗ Very dangerous because it opens security holes for other tenants during the test
kubectl delete networkpolicy secure-database-policy -n production

# CORRECT: Monitor packet drops in real-time using the Cilium CNI monitor
# Step 1: Find the Cilium CNI Pod name running on the application's worker node
kubectl get pods -n kube-system -l k8s-app=cilium -o wide

# Step 2: Run cilium monitor with the drop packet type filter
# (This command detects packet drop reasons along with the policy identity)
kubectl exec -it -n kube-system cilium-xxxxx -- cilium monitor --type drop

The Cilium monitor output gives very precise clarity about the packet drop reason:

xx drop at L3: packet dropped by local policy (egress) flow-id=102345 saddr=10.244.1.45 daddr=10.244.2.12 dport=5432

With this information, we know for certain there’s an egress policy on the sender Pod (10.244.1.45) blocking outgoing traffic to port 5432 on the database (10.244.2.12). We can immediately fix it by writing the right egress NetworkPolicy rules without having to turn off our cluster’s security.

Production Network Diagnostic Checklist #

Use the checklist below as a quick guide when we’re called to handle a network incident in the middle of the night:

POD STATUS & BASIC HEALTH:
  □ Check Pod lifecycle status (`kubectl get pods -o wide`)
  □ Make sure containers aren't restarting or in CrashLoopBackOff status
  □ Verify the application's Readiness Probe status (`kubectl describe pod`)

IP CONNECTIVITY (PING & ROUTING):
  □ Ping the destination Pod IP from a debug container on the same node
  □ Ping the destination Pod IP from a debug container on a different worker node
  □ Test the CNI overlay encapsulation port connectivity (UDP port 4789 / 6081) between hosts

DNS RESOLUTION (CORE_DNS):
  □ Run `nslookup kubernetes.default` from inside the problematic Pod
  □ Check CoreDNS error logs (`kubectl logs -n kube-system -l k8s-app=kube-dns`)
  □ Make sure the NetworkPolicy allows UDP/TCP egress to port 53 in the kube-system namespace
  □ Check the `ndots` option in `/etc/resolv.conf` if external resolution is slow

SERVICE RULES & LOAD FORWARDING:
  □ Make sure the Service has a non-empty Endpoints list (`kubectl get endpoints`)
  □ Match the Service label selector with actual labels attached to Pods
  □ Check the health of kube-proxy pods on worker nodes (`kubectl get pods -n kube-system -l k8s-app=kube-proxy`)

L7 ROUTES & INGRESS CONTROLLER:
  □ Do a port-forward test to the backend Pod directly to validate the app
  □ Check the Service logical port match with the application container target port
  □ Analyze upstream error logs on the Ingress Controller pod (`kubectl logs -n ingress-nginx`)

Summary #

  • Apply gradual isolation techniques: Start network diagnosis from Pod status, then raw IP connectivity, DNS resolution, Service rules, up to L7 Ingress routing rules. Don’t jump to Ingress configuration before making sure the backend IP is reachable.
  • Use Ephemeral Containers for debugging: Leverage the kubectl debug command with the nicolaka/netshoot image so we can access complete diagnostic tools inside the Pod without changing our application’s production image.
  • Monitor Service Endpoints: If the Service returns connection failures, check whether the Endpoints list is empty. This is often caused by applications failing the Readiness Probe or wrong label selector writing.
  • Be wary of ndots’ impact on DNS latency: Kubernetes’ built-in ndots:5 rule forces external domain name lookups to iterate internal queries first. Use a trailing dot (FQDN trailing dot) to speed up resolution.
  • Open CNI encapsulation ports on firewalls: Make sure CNI encapsulation traffic ports (like UDP port 4789 for VXLAN) are open at the physical or cloud provider network firewall level to guarantee smooth cross-node communication.
  • Use CNI instruments to trace drops: Leverage built-in CNI monitoring tools (like cilium monitor) to see packets silently dropped by overly strict NetworkPolicy rules.

← Previous: Ingress Controller Comparison   Next: Networking Anti-Patterns →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact