Network Policy #

In a standard Kubernetes environment, the adopted network model is a flat network that’s open by default (default-open). This means, by default, all Pods in the cluster can communicate with each other directly without any firewall barriers. From a development convenience perspective, this is very pleasant. However, for production cluster environments — especially multi-tenant ones or those hosting various microservices with different data classification levels — this open policy is a security nightmare.

Without restrictions, a vulnerable staging Pod or an insecure development/test Pod can directly send traffic to the production database Pod across namespaces. If one public web container gets exploited by a hacker, that hacker can freely port-scan our entire internal cluster services.

To close this security hole, Kubernetes provides the NetworkPolicy resource. A NetworkPolicy acts as an internal cluster firewall at Layer 3 (IP Address) and Layer 4 (Port/Protocol), declaratively controlled using label selectors.

This article fully dissects how NetworkPolicies work, Zero-Trust security architecture implementation, whitelisting syntax details, the importance of DNS egress rules, and their technical limitations.

How NetworkPolicies Work: A Label-Based Firewall #

Unlike traditional firewalls that require writing rules based on rigid static IP addresses, NetworkPolicies work entirely using the power of Label Selectors. This aligns perfectly with Kubernetes’ dynamic nature where Pod IPs constantly change (ephemeral).

A NetworkPolicy manifest defines rules using three main selection filters:

  • podSelector: Selects which Pods in the local namespace the policy applies to (target pods).
  • ingress: Defines the allowed incoming traffic rules (whitelisted incoming traffic).
  • egress: Defines the allowed outgoing traffic rules (whitelisted outgoing traffic).

Let’s look at the basic traffic filtering architecture by a NetworkPolicy through the following diagram:

flowchart TD
    ClientNormal["Frontend Pod (app=frontend)"] -- "Port 8080 (Allowed)" --> TargetPod["Target Pod (app=api)"]
    ClientRisky["Staging Pod (app=staging)"] -. "Blocked (Not registered)" .-> TargetPod
    TargetPod -- "Port 5432 (Allowed)" --> DB["Database Pod (app=database)"]
    TargetPod -. "Blocked (Port 22)" .-> SSH["SSH Server (Port 22)"]
    
    style ClientRisky fill:#ffcccc,stroke:#ff0000,stroke-width:2px
    style SSH fill:#ffcccc,stroke:#ff0000,stroke-width:2px

1. NetworkPolicies Are Additive #

One thing we must remember is that NetworkPolicies are additive. In Kubernetes, there’s no explicit “DENY” command in rules. All rules are “ALLOW” (whitelisting).

If we create more than one NetworkPolicy targeting the same Pods, the Kubernetes API Server merges all those rules using OR logic. No rule cancels out or overrides another.

2. Policy Enforcement by the CNI #

A NetworkPolicy is just a configuration definition object in the cluster’s etcd database. The Kubernetes API Server doesn’t directly block packets. The enforcement task is fully delegated to the active CNI driver in our cluster.

[!IMPORTANT] Lightweight standard CNIs like Flannel don’t support and ignore all NetworkPolicy manifests we create. If we want to apply NetworkPolicies, we must use a CNI with kernel-level security modules like Calico (using host iptables/IP sets) or Cilium (using the eBPF data path).

The Zero-Trust Security Pattern: Default Deny All #

The best practice for securing production clusters is applying a Zero-Trust approach. We assume all network traffic is dangerous until proven otherwise.

Our first step is applying a Default Deny All policy on our production namespace. This policy closes all ingress and egress ports for all Pods in that namespace. After the doors are tightly closed, we then create specific NetworkPolicy manifests gradually to selectively open access.

The Default Deny All Manifest (Ingress & Egress) #

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-policy
  namespace: production
spec:
  podSelector: {} # {} Empty means targeting ALL Pods in the 'production' namespace
  policyTypes:
    - Ingress
    - Egress
  # Because the ingress and egress blocks are empty, all incoming and outgoing traffic is completely blocked!

Syntax Deep Dive: Whitelisting Ingress & Egress #

After all traffic is blocked, we must open access precisely. Let’s study a NetworkPolicy manifest for isolating the production PostgreSQL database so only our backend application can reach it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-isolation-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres-database # This policy applies specifically to Pods labeled app=postgres-database
  policyTypes:
    - Ingress
  ingress:
    - from:
        # 1. Allow access from Pods labeled app=backend-api in the same namespace
        - podSelector:
            matchLabels:
              app: backend-api
      ports:
        - protocol: TCP
          port: 5432 # Only allow incoming traffic to the PostgreSQL database port

Critical Logic: AND vs OR on Selectors #

One area that most often triggers fatal configuration errors is writing cross-namespace filter syntax. Let’s look at the difference in writing dashes (-) in the following YAML, because they have very different logical meanings:

AND Logic (Namespace AND Pod) #

If we want to restrict access only from Pods labeled app=prometheus inside the namespace labeled kubernetes.io/metadata.name: monitoring, we write both selectors without a new dash separating them:

      ingress:
        - from:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: monitoring
              podSelector: # AND: The Pod must be app=prometheus INSIDE the monitoring namespace
                matchLabels:
                  app: prometheus

OR Logic (Namespace OR Pod) #

If we separate both selectors using a new dash - on each property, we’re using OR logic:

      ingress:
        - from:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: monitoring
            - podSelector: # OR: All Pods in the monitoring namespace CAN enter, OR Pods labeled app=prometheus from ANY namespace can enter
                matchLabels:
                  app: prometheus

The OR syntax above is very dangerous because it unintentionally allows Pods labeled app=prometheus running in insecure staging/dev namespaces to reach our production database.

The Importance of DNS Whitelisting (Port 53) for Egress #

When we apply a Default Deny Egress policy (blocking all outgoing traffic from Pods), we often find our applications suddenly erroring out and unable to connect to any service. This happens because we forgot to allow outgoing permissions for domain name resolution to CoreDNS.

Without egress permission to CoreDNS port 53, our backend Pods can’t learn the IP address of the local cluster database Service or external API domain names (like AWS RDS or the Stripe API).

Example Optimized Egress Manifest (Including DNS Whitelist) #

Here’s a NetworkPolicy manifest for our backend Pods restricting outgoing access only to the PostgreSQL database and the cluster’s CoreDNS for name resolution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-egress-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend-api
  policyTypes:
    - Egress
  egress:
    # 1. Whitelist access to the PostgreSQL database
    - to:
        - podSelector:
            matchLabels:
              app: postgres-database
      ports:
        - protocol: TCP
          port: 5432
    # 2. Whitelist access to CoreDNS (kube-dns) in the kube-system namespace
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Built-in NetworkPolicy Limitations #

Although the built-in Kubernetes NetworkPolicy is very useful, we must understand several of its technical limitations to design appropriate mitigation solutions:

Network FeatureStandard NetworkPolicy SupportCluster Alternative Solutions
DNS Name-Based Filtering (e.g. whitelisting *.stripe.com)Not Supported (only supports manual IP CIDR blocks).Use Ingress/Egress Gateways, Service Meshes (Istio), or Cilium CiliumNetworkPolicy.
Layer 7 Filtering (HTTP Paths, Methods, Headers)Not Supported (only up to the TCP/UDP port level).Implement a Service Mesh (Envoy sidecar proxy) or Cilium L7 Rules.
Packet Log Recording (Auditing dropped packets)Not Supported (no logs when packets are rejected).Use Calico Enterprise audit logs or Cilium’s Hubble CLI.
Policy Precedence OrderingNot Supported (all policies are OR’d without ordering).Use Calico’s GlobalNetworkPolicy objects supporting the order property.

Network Security Anti-Patterns vs Solutions #

Let’s study some fatal mistakes often encountered regarding internal cluster firewall configuration, along with their code comparisons.

Anti-Pattern 1: Leaving the policyTypes Rules Empty on a NetworkPolicy #

We create a NetworkPolicy manifest with an ingress rule inside, but forget to write the policyTypes: [Ingress] property at the top spec.

Wrong Manifest Code (Without Explicit policyTypes) #

# DON'T DO THIS: Policy behavior becomes unpredictable
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: unsafe-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  # policyTypes is not written explicitly!
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend

The Bad Consequences #

If we don’t write policyTypes explicitly, the Kubernetes CNI assumes this policy only applies to the defined things. However, this behavior heavily depends on the Kubernetes version and CNI type we’re running. On some cluster versions, this oversight can make the CNI not apply egress restrictions, so our backend Pods remain free to send data out of the cluster unsupervised.

Solution Code (Always Explicitly Define policyTypes) #

# SOLUTION: Write policyTypes declaratively and clearly
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: safe-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress # Explicitly affirms this policy only locks ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend

Anti-Pattern 2: Using ipBlock to Restrict Inter-Pod Communication #

We want to restrict communication between the frontend Pod and the backend Pod. Because we know the frontend Pod’s current IP is 10.244.1.15, we use the ipBlock property to restrict access to that IP.

Wrong Manifest Code (Restricting Pod Communication with an IP Block) #

# DON'T DO THIS: Pod IPs are dynamic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-ipblock-failure
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.244.1.15/32 # DON'T: The Pod IP changes on restart!

The Bad Consequences #

As soon as the frontend Pod restarts or gets rescheduled due to a rolling update, the new frontend Pod gets a new IP (e.g. 10.244.1.20). Because that IP isn’t registered in the database’s ipBlock rule, the new frontend can’t connect to the database. ipBlock rules may only be used to restrict traffic from/to physical IP addresses outside the cluster (e.g. on-premise database servers or cloud VPC ranges).

Solution Code (Using Standard Label Selectors) #

# SOLUTION: Use podSelector so it dynamically follows IP changes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-selector-success
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend # Safe: The CNI automatically detects new Pod IPs with this label

Practical Guide to Diagnosing NetworkPolicy Disruptions #

If after applying a NetworkPolicy our applications suddenly can’t connect to each other, here are targeted investigation steps:

1. Check the Network Policy Status #

Get a list of all active NetworkPolicies in the target namespace:

kubectl get netpol -n production

Review the selector and rules details attached:

kubectl describe netpol postgres-isolation-policy -n production

2. Test Connectivity Interactively (Netshoot Debugging) #

Run a netshoot debug pod and attach labels so the debug Pod acts as if it’s part of our application Pods:

# Launch a debug Pod with the frontend label to inherit the frontend NetworkPolicy permissions
kubectl run network-probe --rm -i --tty --image nicolaka/netshoot --labels="app=frontend" -n production -- /bin/bash

From inside the network-probe terminal, run a connection test to the database port:

# Test the TCP database port handshake
nc -zvw3 postgres-database 5432

3. Analyze Packet Drop Symptoms #

  • Connection Refused: Means the data packet successfully penetrated the NetworkPolicy firewall, but no application is listening on the target Pod’s port (check the target Pod’s application status).
  • Connection Timeout: Strongly indicates our data packets are being dropped mid-way by the CNI NetworkPolicy rules. Review the ingress rule direction on the target or egress on the sender.

Summary #

  • Apply the Zero-Trust model: Don’t let the cluster run with the default-open model. Apply a Default Deny All policy first, then create whitelist rules gradually.
  • Ensure CNI compatibility: Remember that basic CNIs like Flannel ignore NetworkPolicies. Use advanced CNIs like Calico or Cilium for cluster firewall enforcement.
  • Understand AND vs OR logic: Pay attention to dash (-) writing in YAML manifests. Don’t accidentally use cross-namespace OR logic that can open security holes.
  • Always open CoreDNS access: When applying Default Deny Egress, always include outgoing access whitelisting to port 53 UDP/TCP in the kube-system namespace.
  • Avoid ipBlock for internal Pod communication: ipBlock is only meant for IP addresses outside the cluster. For inter-Pod communication, always rely on the dynamic podSelector.
  • Investigate with probe tools: Use the netshoot debugger pod configured with special labels to simulate smooth data packet flows and detect connection drops.

← Previous: Ingress   Next: Load Balancing & kube-proxy →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact