Network Security #

By default, the Kubernetes network model operates on a very open fundamental principle: a flat network. All Pods in the cluster can freely communicate with each other across namespaces without needing NAT (Network Address Translation) or extra port routing configuration. In development environments, this model makes testing inter-service interactions very easy. However, in production environments, leaving the flat network wide open without restrictions is a nightmare for information security.

If one of our application Pods (e.g. a poorly maintained public blog) gets exploited by an attacker, the flat network model makes it easy for the attacker to do internal network scanning (lateral scanning). The attacker can directly send data packets to transaction databases, Redis caches, or internal control panels that should be isolated. To prevent this, we must apply a Network Zero-Trust architecture in Kubernetes using NetworkPolicies, Mutual TLS (mTLS), cloud provider metadata restrictions, and hardening at the traffic entry point (Ingress).

1. The Network Zero-Trust Philosophy in Kubernetes #

Network Zero-Trust is based on three simple postulates: never trust, always verify, and restrict network access rights as minimally as possible. In Kubernetes, this means we must break the assumption that internal cluster data traffic is safe.

Network Security Pillars: #

  • L3/L4 Segmentation (NetworkPolicy): Restricts inter-Pod connections based on IP addresses, label selectors, namespaces, and TCP/UDP ports.
  • L7 Identity & Encryption (mTLS): Guarantees data sent between Pods is cryptographically encrypted on the wire and only processes with legitimate identities (SPIFFE/SPIRE) can open connections.
  • Perimeter Security (Ingress Hardening): Filters public internet traffic before it enters the internal cluster network.

2. Network Segmentation with NetworkPolicies (L3/L4) #

NetworkPolicy is a declarative Kubernetes spec acting as a local firewall for Pods. Note that the Kubernetes API Server only stores NetworkPolicy object definitions. The task of evaluating and enforcing those data packet rules is fully delegated to the CNI (Container Network Interface) Plugin we install in the cluster (like Calico, Cilium, or Kube-Router).

[!WARNING] CNIs Without Policy Support: If we use a basic CNI like Flannel, all NetworkPolicy objects we create are silently ignored. Flannel has no data packet evaluation engine (iptables/eBPF), so traffic stays wide open. Always use a security-supporting CNI like Calico or Cilium in production.

Stage 1: Apply Default-Deny-All #

When designing firewalls, the safest tactic is blocking all traffic first (default-deny), then selectively opening access (allow-list).

# Default Deny All Manifest for Ingress and Egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: e-commerce # Apply to the production namespace
spec:
  # Using empty curly braces '{}' to match all Pods in the namespace
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Stage 2: Open DNS Access (Mandatory Egress) #

After enabling the default-deny Egress above, all application domain name resolution (DNS) queries to the cluster’s CoreDNS (kube-dns) get blocked. Our applications immediately become paralyzed because they can’t find the database IP addresses or external services.

Therefore, we must create a special rule allowing outgoing (Egress) traffic to CoreDNS on port 53 (UDP/TCP).

# Manifest Allowing CoreDNS Resolution Access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-system-dns
  namespace: e-commerce
spec:
  podSelector: {} # Applies to all Pods in the namespace
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          # Targets the 'kube-system' namespace where CoreDNS runs
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Stage 3: Open Inter-Service Ingress Access #

Let’s say we have a payment-api microservice that may only accept incoming connections (Ingress) from the gateway-api microservice in the same namespace.

# Manifest Restricting Ingress to payment-api
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-payment-from-gateway
  namespace: e-commerce
spec:
  # Targets the target Pod (payment-api)
  podSelector:
    matchLabels:
      app: payment-api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          # Only allow incoming connections from Pods with this label
          app: gateway-api 
    ports:
    - protocol: TCP
      port: 8080 # Restrict access to only this application port

3. Preventing Credential Leaks: Block Cloud Provider Metadata #

When we run Kubernetes in cloud provider environments (like AWS EKS, GCP GKE, or Azure AKS), every worker node has physical access to the cloud’s local metadata endpoint via the link-local IP address 169.254.169.254.

If one of our applications is vulnerable to SSRF (Server-Side Request Forgery), attackers can exploit that application to send HTTP GET queries to that metadata endpoint.

[ Infected Application Pod ] ──► HTTP GET http://169.254.169.254/latest/meta-data/iam/security-credentials/
                                                  │
                                                  ▼
                        [ Returns the Production IAM Role Key Token ]

Attackers obtain the IAM Role access token attached to the worker node, giving them the ability to steal S3 bucket data, manipulate cloud databases, or even delete the entire cluster infrastructure.

The NetworkPolicy Egress Solution #

We must create a global-level NetworkPolicy prohibiting all container Pods from accessing the 169.254.169.254 IP.

# Manifest Blocking Access to the Cloud Provider Metadata Endpoint
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-cloud-metadata
  namespace: e-commerce
spec:
  podSelector: {} # Apply to all Pods in the namespace
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        # Allows outgoing connections to all internet IP addresses (0.0.0.0/0)
        cidr: 0.0.0.0/0
        except:
        # EXCEPT to these sensitive cloud provider metadata IPs
        - 169.254.169.254/32 # AWS / GCP / Azure Metadata IP
        - 169.254.170.2/32   # AWS ECS Task Metadata IP

4. Layer 7 Authorization & Encryption: Mutual TLS (mTLS) #

NetworkPolicies work exclusively at Layer 3 (IP) and Layer 4 (Port). L3/L4 networks don’t have the ability to:

  1. Detect whether data packets are sent in sniffable plaintext on host node network switches.
  2. Validate the real identity of the sending container (attackers can manipulate IP spoofing at a compromised CNI level).

To secure data traffic at the application level, we must enable Mutual TLS (mTLS) using Service Mesh technology (like Istio or Linkerd) or at the modern CNI level (like Cilium).

mTLS guarantees every inter-Pod connection goes through a two-way TLS handshake. The sending Pod validates the receiving Pod’s certificate, and vice versa. Additionally, all data traffic is automatically encrypted using modern encryption algorithms (AES-GCM).

sequenceDiagram
    participant PodA as Gateway Pod (Client)
    participant ProxyA as Envoy Proxy A (Sidecar)
    participant ProxyB as Envoy Proxy B (Sidecar)
    participant PodB as Payment Pod (Server)
    
    PodA->>ProxyA: Plaintext HTTP Request (e.g., POST /pay)
    ProxyA->>ProxyB: TLS Handshake (Send the SPIFFE Client Certificate)
    ProxyB->>ProxyA: TLS Handshake (Send the SPIFFE Server Certificate)
    Note over ProxyA,ProxyB: Verify Two-Way Certificate Validity (mTLS)
    ProxyA->>ProxyB: Send Encrypted Data (AES)
    ProxyB->>PodB: Forward the Plaintext HTTP Request (Localhost)

If we use Istio, we can enforce a strict mTLS policy (strict mode) across the entire production namespace with one manifest:

# Istio STRICT mTLS Enforcement Manifest
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default-mtls-strict
  namespace: e-commerce
spec:
  mtls:
    # STRICT mode rejects all plaintext non-mTLS connections trying to enter the namespace
    mode: STRICT

5. Ingress Hardening (Security Perimeter) #

The Ingress Controller is our main defense gateway against public internet traffic threats. We must configure security hardening on the Ingress to mitigate common attacks like brute force, payload injection, and DDoS.

Here’s the recommended NGINX Ingress annotation configuration for production:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gateway-ingress
  namespace: e-commerce
  annotations:
    # 1. Limit Request Speed per IP (Rate Limiting)
    nginx.ingress.kubernetes.io/limit-rps: "50" # Maximum 50 requests per second per client IP
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    
    # 2. Force HTTPS with HTTP Strict Transport Security (HSTS)
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/hsts: "true"
    nginx.ingress.kubernetes.io/hsts-max-age: "31536000" # Enforce HTTPS for 1 year
    nginx.ingress.kubernetes.io/hsts-include-subdomains: "true"
    
    # 3. Limit the Request Payload Size (Buffer Overflow Mitigation)
    nginx.ingress.kubernetes.io/proxy-body-size: "5m" # Reject requests with body > 5 MB
    
    # 4. Inject HTTP Security Headers (Configuration Snippet)
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Clickjacking Protection
      more_set_headers "X-Frame-Options: DENY";
      # MIME Sniffing Protection
      more_set_headers "X-Content-Type-Options: nosniff";
      # Controls the referer information sent
      more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
      # Refuses access to client hardware devices
      more_set_headers "Permissions-Policy: camera=(), microphone=(), geolocation=()";      
spec:
  ingressClassName: nginx
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: gateway-service
            port:
              number: 80

6. The Network Packet Evaluation Decision Flow (CNI Engine) #

How the CNI evaluates every data packet crossing container network interfaces can be logically described through the following flowchart:

flowchart TD
    Start["Data Packet Sent / Received on the Node"] --> Q1{"Is there an active NetworkPolicy<br>targeting the Pod?"}
    
    Q1 -- "No" --> AllowDefault["ALLOW THE PACKET (Flat Network Default)"]
    Q1 -- "Yes" --> Q2{"Does the packet origin / destination match<br>the ALLOW-LIST rules in the NetworkPolicy?"}
    
    Q2 -- "Yes (A Matching Rule Exists)" --> AllowPolicy["ALLOW THE PACKET"]
    Q2 -- "No (No Match)" --> DropPacket["BLOCK & DROP THE PACKET<br>'(Silent Drop)'"]
    
    style AllowDefault stroke:#f57c00,stroke-width:2px
    style AllowPolicy stroke:#388e3c,stroke-width:2px
    style DropPacket stroke:#d32f2f,stroke-width:2px

Anti-Patterns vs Best Solutions #

Here are common network security configuration mistakes in Kubernetes clusters along with their best fixes.

Anti-Pattern 1: Enabling Ingress for Database Servers #

Exposing the cluster’s main database (like PostgreSQL or MySQL) using a LoadBalancer-type Service or TCP Ingress so developer teams can query directly from their laptops.

Consequences #

The database is publicly exposed to the entire internet. This invites mass brute force attacks and increases data leak risks if a zero-day vulnerability exists in the database engine.

Best Solution #

Keep the database Service type as ClusterIP (only accessible internally in the cluster). If developer teams need emergency debugging access, use a Bastion Host mechanism or encrypted port-forwarding via VPN / RBAC.

# ✓ SOLUTION: Use local encrypted port-forwarding via Kubernetes RBAC
kubectl port-forward svc/postgres-db-service 5432:5432 -n database
# Developers access the database locally at 'localhost:5432' via the kubectl TLS tunnel

Anti-Pattern 2: Default-Deny Egress Without Opening the DNS Port #

Applying a strict default-deny Egress NetworkPolicy on the production namespace, but forgetting to specifically open UDP port 53 to CoreDNS.

Consequences #

All applications in that namespace suffer CrashLoopBackOff because they fail DNS lookups to reach databases or external APIs.

Best Solution #

Always install the allow-system-dns manifest (like the example at the top of this article) in every namespace applying default-deny Egress.

Complete Production Zero-Trust Network Manifests #

Here’s a combined production-ready manifest example locking down the network security of the checkout-api microservice inside the e-commerce namespace:

# 1. DEFAULT-DENY-ALL rule for initial isolation
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-default-deny
  namespace: e-commerce
spec:
  podSelector:
    matchLabels:
      app: checkout-api
  policyTypes:
  - Ingress
  - Egress
---
# 2. ALLOW-INGRESS rule: Only allow incoming traffic from gateway-api
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow-ingress
  namespace: e-commerce
spec:
  podSelector:
    matchLabels:
      app: checkout-api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: gateway-api
    ports:
    - protocol: TCP
      port: 8080
---
# 3. ALLOW-EGRESS rule: Only allow outgoing traffic to CoreDNS and the Redis database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow-egress
  namespace: e-commerce
spec:
  podSelector:
    matchLabels:
      app: checkout-api
  policyTypes:
  - Egress
  egress:
  # A. Access to CoreDNS (Kube-System Namespace)
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # B. Access to the Redis Database (Redis Pod in the same namespace)
  - to:
    - podSelector:
        matchLabels:
          app: redis-cache
    ports:
    - protocol: TCP
      port: 6379

Zero-Trust Network Review Checklist #

Use the following checklist to verify the security of our production cluster network architecture:

SEGMENTATION & NETWORKPOLICY:
  □ The CNI plugin used in production is verified to support NetworkPolicies (e.g. Calico, Cilium).
  □ The 'default-deny-all' policy is applied to all application namespaces.
  □ The 'allow-system-dns' rule is successfully configured in every default-deny namespace.
  □ Multi-tenant inter-namespace traffic segmentation runs strictly (isolated).

L7 ENCRYPTION & AUTHENTICATION:
  □ STRICT mTLS is enabled using a Service Mesh (Istio/Linkerd) across all sensitive namespaces.
  □ Transport network encryption (WireGuard/IPSec) is enabled at the node-to-node level (CNI Level).
  □ Internal database connections are confirmed to use encrypted SSL/TLS ports.

ENDPOINT & CLOUD PORT PROTECTION:
  □ Pod outgoing access to the cloud provider metadata IP (169.254.169.254) is explicitly blocked.
  □ The Ingress Controller is configured with RPS rate-limiting rules and proxy-body-size limits.
  □ Security HTTP Headers (X-Frame-Options, HSTS, Content-Type-Options) are dynamically injected on the Ingress.

Summary #

  • Block the flat network default — Turn off the Kubernetes default behavior letting all Pods communicate by applying strict NetworkPolicy segmentation.
  • Must install a security CNI — Don’t use Flannel in production because it doesn’t evaluate NetworkPolicy manifests; choose Calico or Cilium supporting cluster firewall enforcement.
  • Apply default-deny-all — Start network security by blocking all Ingress & Egress, then open access specifically and granularly to minimize human error.
  • Isolate cloud metadata IPs — Protect the cluster from IAM token theft attacks (SSRF) by blocking Pod outgoing access to the cloud provider link-local IP 169.254.169.254.
  • Enforce STRICT mTLS — Use a Service Mesh to guarantee inter-Pod data traffic is automatically encrypted and authenticated using cryptographic identities.
  • Harden Ingress annotations — Protect the cluster’s outer perimeter by enabling rate limiting, payload size limits, HSTS redirection, and security HTTP headers on the Ingress.

← Previous: Pod Security   Next: Supply Chain Security →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact