Networking Anti-Patterns #

Networking in Kubernetes adopts a very different model compared to traditional comparison environments like standalone virtual machines (VMs) or standard Docker container handling. While Docker uses a host port mapping model with dynamic NAT allocation, Kubernetes implements a flat network model where every Pod gets a unique IP address directly reachable from any node without address translation help.

This abstraction ease often makes us careless. Network problems in Kubernetes are very prone to silent failures — services look normal during local or staging testing phases with low load, but systematically fail when facing highly fluctuating production traffic. This article summarizes the eight most common networking anti-patterns in production environments, their technical consequences, and more robust alternative solutions.

1. Avoid Hardcoding Pod IPs for Inter-Service Communication #

Pod IPs in Kubernetes are ephemeral. Every time a Pod restarts due to an application failure, a rolling update process, or when a worker node gets drained for maintenance, that Pod is destroyed and recreated with a new IP from the node’s CIDR subnet.

Technical Consequences #

If we configure a microservice (e.g. order-service) to contact a PostgreSQL database by registering the database Pod’s physical IP directly in a configuration file, connectivity breaks instantly when that database Pod gets rescheduled. Our application shows connection refused or timeout errors, and we’re forced to manually update configuration and redeploy in the middle of the night.

Implementation Comparison #

# ANTI-PATTERN: Registering the database Pod IP directly in application code
# If the database Pod restarts, the IP address 10.244.2.34 is no longer valid.
DATABASE_HOST = "10.244.2.34"
DATABASE_PORT = 5432

# CORRECT: Use the stable Kubernetes Service domain name
# This domain is dynamically resolved to a constant ClusterIP by CoreDNS.
DATABASE_HOST = "postgres-db.production.svc.cluster.local"
DATABASE_PORT = 5432

2. Avoid Creating One LoadBalancer Service for Every Microservice #

When we need to expose internal applications to the outside internet, Kubernetes provides a Service type called LoadBalancer. However, creating Service objects with this spec for every microservice is an inefficient architectural approach.

Technical Consequences #

In public cloud environments (like AWS, GCP, or Azure), every time the API Server detects a LoadBalancer-type Service, it asks the cloud provider API to instantiate one new physical Load Balancer unit (e.g. an AWS Network Load Balancer or a GCP Layer 4 Load Balancer). If our cluster has 30 internal microservices all exposed this way:

  1. Ballooning Costs: Cloud providers charge per hour for every physical load balancer instance created.
  2. Complicated SSL/TLS Management: We must configure SSL/TLS certificates separately on every load balancer.
  3. Public IP Limitations: We unnecessarily exhaust our VPC public IP quota.

Implementation Comparison #

# ANTI-PATTERN: Exposing every microservice with a separate LoadBalancer
# ✗ This creates a new physical load balancer for every manifest.
apiVersion: v1
kind: Service
metadata:
  name: billing-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: billing
---
# CORRECT: Use the ClusterIP Service type and manage external routes with a single Ingress object
# ✓ Only one physical Load Balancer is created (in front of the Ingress Controller) to split traffic.
apiVersion: v1
kind: Service
metadata:
  name: billing-service
spec:
  type: ClusterIP  # Only accessible from inside the cluster
  ports:
  - port: 80
  selector:
    app: billing
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-secret
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /billing
        pathType: Prefix
        backend:
          service:
            name: billing-service
            port:
              number: 80

3. Avoid Letting Pods Enter Endpoints Without Readiness Probes #

By default, if we don’t specify a readinessProbe configuration in the Deployment manifest, Kubernetes considers the application container ready to receive incoming traffic as soon as the container status is recorded as Running.

Technical Consequences #

During a new deployment process (rolling update), Kubernetes creates the new version Pod and immediately adds it to the Service’s Endpoints list as soon as its container process starts. In reality, modern web applications often need time (between 5 and 30 seconds) to initialize frameworks, load configuration from a secret manager, or open database connection pools.

As a result, HTTP query traffic from users gets routed to new Pods that aren’t ready to process requests. Our users get 502 Bad Gateway or 503 Service Unavailable error responses in every application release cycle.

Implementation Comparison #

# ANTI-PATTERN: Deployment without a readiness check definition
# ✗ The Pod immediately receives traffic once the container OS process runs.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-unsafe
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-app:v2.0.0
        ports:
        - containerPort: 8080
---
# CORRECT: Use a readinessProbe to verify the app is ready to process queries
# ✓ The Pod is only added to the Service Endpoints after the /healthz/ready endpoint responds with HTTP 200.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-safe
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-app:v2.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 5  # Wait 5 seconds before starting to monitor
          periodSeconds: 5        # Check every 5 seconds
          failureThreshold: 3     # Remove from Endpoints if it fails 3 times in a row

4. Avoid Using NodePort for Public Exposure in Production Environments #

A NodePort-type Service is the easiest way for beginners to access cluster applications from outside, because it opens one static high-value port (in the default 30000–32767 range) on all cluster worker nodes.

Technical Consequences #

Although functional, using NodePort in production environments facing the public internet is an anti-pattern endangering our cluster security:

  1. Wide-Open Security Holes: High network ports are directly open on every public IP of our worker nodes.
  2. No Built-in TLS/SSL Features: NodePort doesn’t provide automatic gateway-level TLS encryption.
  3. Node IP Dependency (High Vulnerability): External users must know the worker node’s physical IP to connect. If that node dies, user connections break unless they manually switch the connection target to another worker node IP.

Public Route Handling Comparison #

ANTI-PATTERN (Public NodePort Exposure):
  [User] ---> (Direct Connection to Port 31080) ---> [Worker-Node-IP-1:31080] 
  (Risk: Public ports open on all nodes, no centralized SSL)

CORRECT (Protected Ingress / LoadBalancer Exposure):
  [User] ---> (Port 443 HTTPS) ---> [Cloud Load Balancer] ---> [Ingress Controller] ---> [ClusterIP Service]
  (Security: Centralized TLS encryption at the Ingress, Node IPs hidden inside a private VPC)

NodePort should only be used as an internal bridge behind an external Load Balancer system (especially on on-premise clusters) or for quick local testing on our development computers.

5. Avoid Applying Default-Deny NetworkPolicies Without Opening DNS Egress #

Applying a zero-trust security policy by creating a global default-deny NetworkPolicy (both ingress and egress) is an excellent security mitigation step for isolating traffic between namespaces. However, a fatal mistake often happens when we forget to open the outbound path for domain name resolution (DNS).

Technical Consequences #

When the default-deny rule blocks all outgoing (egress) traffic from a namespace, DNS queries from application Pods to CoreDNS port 53 UDP/TCP in the kube-system namespace also get silently dropped by the Node kernel firewall system.

As a result, applications can’t resolve external domain names (like third-party payment APIs) or internal cluster domain names. Our applications stop completely with confusing unknown host or connection timeout error messages.

Implementation Comparison #

# ANTI-PATTERN: Total egress default-deny without exceptions
# ✗ All domain name queries are completely blocked, paralyzing the app.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-all-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
---
# CORRECT: Apply default-deny but include port 53 egress permission to CoreDNS
# ✓ The app stays safe from external traffic leaks, but domain name resolution still works.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-with-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  # MANDATORY: Allow internal DNS queries to the kube-system namespace on port 53
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # Add other specific egress rules below (e.g. to the database)
  - to:
    - podSelector:
        matchLabels:
          app: postgres-db
    ports:
    - protocol: TCP
      port: 5432

6. Avoid externalTrafficPolicy: Cluster If You Need the Original Client IP #

By default, LoadBalancer- or NodePort-type Services use the externalTrafficPolicy: Cluster spec. When a data packet from an external user enters Worker Node A, kube-proxy receives that packet and randomly routes it to one of the backend Pods, which might be on Worker Node B.

Technical Consequences #

To move the packet across nodes, Worker Node A must perform Source Network Address Translation (SNAT). As a result, the source IP on the data packet gets changed to Worker Node A’s internal IP.

When the packet arrives at the application Pod, our application sees all incoming query traffic coming from the cluster node IP, not the user’s real public IP. This triggers several crucial losses:

  1. Failed Log Analysis: We lose the original IP trail for security audit or incident forensics needs.
  2. Broken Geo-Blocking: The app can’t detect the client’s country of origin for content filtering.
  3. Failed Limiting Security: IP-based rate limiting systems can’t work correctly because all users are considered to have the same IP.

Packet Routing Flow and Source IP Modification #

Let’s study the packet routing visualization below to see how the original client IP gets modified by SNAT rules under the default Cluster policy:

flowchart TD
    subgraph "externalTrafficPolicy: Cluster (Default)"
        direction TB
        ClientA["Client (Original IP: 203.0.113.5)"] -->|Packet| LBA["Cloud Load Balancer"]
        LBA --> NodeA["Worker Node A (No Pod)"]
        NodeA -->|"SNAT (Source IP changed to Node A's IP: 10.0.1.10)"| NodeB["Worker Node B (Has Pod)"]
        NodeB --> PodB["Backend Pod (Sees Sender IP: 10.0.1.10)"]
    end

    subgraph "externalTrafficPolicy: Local"
        direction TB
        ClientB["Client (Original IP: 203.0.113.5)"] -->|Packet| LBB["Cloud Load Balancer"]
        LBB -->|"Only direct to nodes with Pods"| NodeD["Worker Node D (Has Pod)"]
        NodeD -->|"No SNAT"| PodD["Backend Pod (Sees Original IP: 203.0.113.5)"]
    end

Service Configuration Comparison #

# ANTI-PATTERN: Using the 'Cluster' default value when Client IP verification is needed
# ✗ Kube-proxy does cross-node SNAT, erasing the original sender IP data.
apiVersion: v1
kind: Service
metadata:
  name: app-service-no-ip
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster  # The default value
  ports:
  - port: 80
  selector:
    app: web
---
# CORRECT: Set externalTrafficPolicy to 'Local'
# ✓ Packets from the Load Balancer are only sent to nodes hosting active backend Pods without SNAT manipulation.
apiVersion: v1
kind: Service
metadata:
  name: app-service-preserve-ip
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local  # Preserves the original Client IP
  ports:
  - port: 80
  selector:
    app: web

[!WARNING] Setting externalTrafficPolicy: Local can cause uneven load distribution if Pod replica counts aren’t spread evenly across every Worker Node, because the cloud Load Balancer only sends traffic to Worker Nodes hosting active target Pods.

7. Avoid Ignoring Consistent Named Logical Ports #

Repeatedly writing port configuration using raw integer port numbers across Pod, Service, and Ingress manifests without giving them logical name labels (named ports) is one of the configuration management anti-patterns most often triggering port mapping inconsistencies.

Technical Consequences #

When a backend developer changes the application’s listening port in code from 8080 to 9000 (e.g. due to a container runtime framework change), we must find and replace all those port numbers across dozens of different YAML manifest lines.

If one Service manifest is missed in target port alignment, we face an upstream routing error condition (502 Bad Gateway) in production. Additionally, the lack of standardized port names makes integrating external tools like the Istio Service Mesh or Prometheus monitoring systems harder for dynamically detecting scrape ports.

Manifest Port Writing Comparison #

# ANTI-PATTERN: Using raw port numbers scattered across many manifests
# ✗ Very prone to typos if the backend application port changes.
apiVersion: v1
kind: Pod
metadata:
  name: web-app-raw-port
spec:
  containers:
  - name: app
    image: my-app:v1
    ports:
    - containerPort: 8080  # Raw port number
---
apiVersion: v1
kind: Service
metadata:
  name: app-service-raw
spec:
  ports:
  - port: 80
    targetPort: 8080  # Must be manually aligned with the containerPort above
  selector:
    app: web
---
# CORRECT: Define Named Ports at the Pod container spec level
# ✓ Services and Ingresses just refer to that name label, free from physical port number coupling.
apiVersion: v1
kind: Pod
metadata:
  name: web-app-named-port
  labels:
    app: web
spec:
  containers:
  - name: app
    image: my-app:v1
    ports:
    - name: http-web-port  # Define the port name here
      containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: app-service-named
spec:
  ports:
  - port: 80
    targetPort: http-web-port  # Refer to the port name logically
  selector:
    app: web

8. Avoid DNS Query Leaks to the Outside from an Oversized ndots Configuration #

By default, every Pod in Kubernetes is configured with the ndots:5 parameter value in its /etc/resolv.conf DNS resolution file.

Technical Consequences #

The ndots:5 parameter instructs the container’s internal resolver system that if the queried domain name has fewer than 5 dot characters (.), the resolver must first look up that domain by appending the cluster’s internal search domain suffixes, before trying to resolve it as an absolute external domain.

If our application frequently makes HTTP calls to external domains like api.stripe.com (which only has 2 dots):

  1. The resolver tries api.stripe.com.production.svc.cluster.local (Failed/NXDOMAIN).
  2. The resolver tries api.stripe.com.svc.cluster.local (Failed/NXDOMAIN).
  3. The resolver tries api.stripe.com.cluster.local (Failed/NXDOMAIN).
  4. Finally the resolver tries api.stripe.com absolutely (Success).

For every external query call, our resolver does 3 wasted DNS queries to CoreDNS. Under high production load, this triggers DNS query overload that can spike CoreDNS CPU to 100% and slow down our entire cluster application latency.

External Query Handling Comparison #

# ANTI-PATTERN: Calling standard external domains without a trailing dot
# ✗ This makes CoreDNS do 3 wasted internal queries first due to the ndots rule.
response = requests.get("https://api.stripe.com/v3/charges")

# CORRECT: Use a closing dot (FQDN Trailing Dot) on external domains
# ✓ The trailing dot tells the resolver this domain is absolute, skipping internal search queries.
response = requests.get("https://api.stripe.com./v3/charges")

We can also reduce the ndots value centrally on the Pod spec manifest if our application never uses short cross-namespace domain calls:

# Alternative mitigation: Reduce the ndots value in the Pod manifest
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"  # DNS only appends the search path if the domain has no dots at all

Cluster Network Review Checklist #

Use the worksheet below to audit our manifests’ network configuration before releasing to production:

ROUTE DIRECTION & SERVICE STRUCTURE:
  □ All application Pods must be protected by at least one readinessProbe
  □ All internal Services use the ClusterIP type, not NodePort/LoadBalancer
  □ Port mappings in manifests use consistent logical names (named ports)
  □ External domain names in application code are written with a trailing dot

SECURITY & ISOLATION (NETWORK_POLICY):
  □ Namespaces are protected by default-deny policies to minimize lateral attack paths
  □ Every default-deny NetworkPolicy must include port 53 UDP/TCP permission to the kube-system namespace
  □ External database CIDR IPs are protected with precise egress rules

IDENTITY PRESERVATION (SOURCE_IP):
  □ LoadBalancer Services holding public traffic logs use externalTrafficPolicy: Local
  □ Ingress Controller annotation configuration is aligned with the X-Forwarded-For header parameter

Summary #

  • Abstract Pod IP addresses: Avoid writing Pod IP addresses statically in application code. Always use the internal cluster Service DNS name that’s guaranteed stable and dynamic by CoreDNS.
  • Save Load Balancer costs: Use one single Ingress Controller gateway to split traffic to dozens of internal cluster ClusterIP Services rather than wasting cloud costs creating separate LoadBalancer Services for every application.
  • Prevent release failures with readinessProbes: Make sure all containers have readinessProbes so Kubernetes doesn’t rush sending traffic to new Pods still in internal initialization.
  • Protect nodes by disabling NodePort: Don’t expose NodePort directly to the public internet. Keep centralized routing through the Ingress Controller for security and TLS termination flexibility reasons.
  • Maintain the CoreDNS query path: When designing zero-trust architectures with NetworkPolicies, make sure port 53 UDP/TCP to the CoreDNS namespace is excluded so applications don’t lose domain name resolution ability.
  • Consciously preserve the original Client IP: Use the externalTrafficPolicy: Local parameter on LoadBalancer Services to avoid SNAT manipulation by kube-proxy, so our applications can process real external user IPs.
  • Reduce CoreDNS load with trailing dots: Add a closing dot on external domain address calls in our applications to avoid wasted internal cluster search queries from the default ndots:5 rule.

← Previous: Network Troubleshooting   Next: ConfigMap →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact