Ingress #

When we design web or API application exposure outside the cluster, we’re introduced to the LoadBalancer-type Service. That type is very reliable for exposing one external service. However, as our system grows into a microservices architecture, we find ourselves with dozens of different backend services (e.g. user, payment, dashboard, catalog services, etc.).

If we create one LoadBalancer Service for each of those services, we’re forced to pay very expensive monthly costs for dozens of physical Load Balancers provided by the cloud provider. Additionally, managing dozens of different public IP addresses or external public DNS names becomes an operational nightmare for SysOps teams.

To solve this cost efficiency and layer-7 routing flexibility problem, Kubernetes provides a special object called Ingress. Ingress acts as a single unified entry point receiving all external HTTP/HTTPS traffic into the cluster, then distributing it to the right internal services based on host domain name rules (host-based routing) or URL path rules (path-based routing).

This article deeply reviews Ingress components, how the Ingress Controller works, writing manifest rules, TLS installation, production annotation optimization, and the evolution toward the Gateway API.

Ingress Components: Resource vs Controller #

One important concept we must understand when first using Ingress is the separation between configuration and implementation. Ingress in Kubernetes is divided into two separate components:

flowchart TD
    System["KUBERNETES INGRESS SYSTEM"]
    System --> Resource["Ingress Resource (YAML manifest defining routing rules)"]
    System --> Controller["Ingress Controller (Active reverse proxy app processing traffic)"]

1. The Ingress Resource #

This is a standard Kubernetes YAML manifest we create to define traffic routes. Inside this manifest, we write rules like: “If incoming traffic has the domain api.example.com going to URL path /payments, redirect that traffic to Service payment-service on Port 80”. This object is just static configuration data in the etcd database; it doesn’t process any data itself.

2. The Ingress Controller #

The Ingress Controller is an active reverse proxy and load balancer application running as a Pod inside our cluster. Kubernetes doesn’t include an Ingress Controller by default when a cluster initializes. We must install it manually ourselves (e.g. installing the NGINX Ingress Controller, Traefik, Kong, or HAProxy).

The Ingress Controller watches the Kubernetes API Server for every Ingress Resource object we create. Once it detects new configuration or route changes, the Ingress Controller dynamically updates its internal reverse proxy configuration file and reloads (e.g. rewriting the nginx.conf block at runtime) without disconnecting active users.

The Ingress Request Handling Flow #

Let’s look at how data packets from outside the cluster flow through the Ingress components in the following diagram:

flowchart TD
    Client["External client (api.company.com/payments)"] --> LB["Cloud Load Balancer (NLB/ALB)"]
    LB --> IngressCtrl["Ingress Controller Pod (NGINX/Envoy)"]
    IngressCtrl -. Evaluate rules .-> Svc["payment-service (ClusterIP)"]
    Svc --> Pod1["Pod payment-0"]
    Svc --> Pod2["Pod payment-1"]

When an external client sends an HTTPS query to api.company.com/payments:

The connection lands on our one physical external Cloud Load Balancer.
The Load Balancer forwards the connection to the active Ingress Controller Pod in the cluster.
The Ingress Controller analyzes the query’s HTTP headers (finding host api.company.com and path /payments).
The Ingress Controller evaluates rule matches, then bypasses the internal ClusterIP Service and sends that traffic directly (direct bypass) to one healthy payment backend Pod IP registered in the Endpoints table.

Production Ingress Manifest Structure #

Let’s study an example standard production Ingress manifest combining host-based routing, path-based routing, and TLS termination:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: enterprise-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx" # Specifies the controller type (legacy method)
    nginx.ingress.kubernetes.io/ssl-redirect: "true" # Redirects HTTP to HTTPS automatically
spec:
  ingressClassName: nginx # Modern standard option for pointing to the Ingress Controller
  tls:
    - hosts:
        - api.mycompany.com
        - app.mycompany.com
      secretName: wildcard-tls-secret # Secret containing the SSL/TLS certificate
  rules:
    # 1. Host-based routing (api.mycompany.com)
    - host: api.mycompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gateway-api-service
                port:
                  number: 8080
    # 2. Path-based routing (app.mycompany.com)
    - host: app.mycompany.com
      http:
        paths:
          - path: /api/users
            pathType: Prefix
            backend:
              service:
                name: user-service
                port:
                  number: 80
          - path: /api/billing
            pathType: Exact
            backend:
              service:
                name: billing-service
                port:
                  number: 80

Diving into the `pathType` Parameter #

Kubernetes provides three URL path matching options we must specify precisely:

Exact: The client’s URL query must match the written path character-for-character. For example, /api/billing only matches /api/billing. Queries like /api/billing/details or /api/billing/ produce a 404 (Not Found) error.
Prefix: The URL query matches if it has the corresponding path prefix. For example, /api/users matches /api/users, /api/users/, /api/users/123, and /api/users/profile. This option is the most common for API routing.
ImplementationSpecific: The matching algorithm is fully delegated to the internal implementation of the Ingress Controller we use. Behavior can differ between controller providers.

TLS Termination and HTTPS Security #

One of Ingress’s best features is its ability to do TLS Termination. With TLS termination, the heavy SSL/TLS cryptographic handshake process is fully delegated at the Ingress Controller level.

Data traffic from the outside internet to the Ingress Controller runs on secure encrypted port 443 HTTPS. After being decrypted at the Ingress Controller, the traffic is forwarded to internal cluster Pods using the regular HTTP port (port 80). This greatly saves CPU power on our application Pods because they don’t need to repeatedly do encryption/decryption processes.

1. Creating the TLS Secret Manually #

We must store our SSL certificate files (.crt or .pem) and private key (.key) as a Kubernetes Secret object of type kubernetes.io/tls:

kubectl create secret tls wildcard-tls-secret \
  --cert=path/to/fullchain.pem \
  --key=path/to/privkey.pem \
  -n production

2. Automatic Integration with cert-manager (Let’s Encrypt) #

Instead of manually renewing SSL certificates every three months, we’re highly advised to install the cert-manager addon. cert-manager collaborates with Let’s Encrypt using the ACME protocol to automate the creation, issuance, and renewal of our SSL certificates.

We just add a special annotation to our Ingress Resource:

apiVersion: v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auto-ssl-ingress
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # Triggers cert-manager to auto-create SSL
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.mycompany.com
      secretName: app-tls-auto-secret # cert-manager writes the Let's Encrypt certificate to this Secret
  rules:
    - host: app.mycompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-service
                port:
                  number: 80

NGINX Ingress Annotations Optimization for Production #

Because the basic Kubernetes Ingress spec is very simple, all advanced reverse proxy configurations (like rate limiting, CORS, rewrites, etc.) are handled using annotations specific to the Ingress Controller we choose.

Here are important annotations for the NGINX Ingress Controller we often need in production:

1. Rate Limiting (DDoS & Brute Force Prevention) #

Limits the number of connection requests from one client IP per second to secure backend availability:

nginx.ingress.kubernetes.io/limit-rps: "15" # Maximum 15 requests per second from one client IP
nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"

2. Allowing Large File Uploads #

By default, NGINX caps the maximum request body size at 1MB. If our application has an image or document upload feature, we must raise this limit to avoid 413 Request Entity Too Large errors:

nginx.ingress.kubernetes.io/proxy-body-size: "50m" # Raises the file upload limit to 50 MegaBytes

Allows frontend applications on other domains (e.g. mobile apps or partner APIs) to query the cluster API safely:

nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "https://my-partner-domain.com"

4. Connection Timeout Configuration #

Prevents connections from hanging too long when processing heavy database queries:

nginx.ingress.kubernetes.io/proxy-read-timeout: "120" # Read timeout limit set to 120 seconds
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"

Ingress vs Gateway API: The Future of Network Routing #

Although Ingress is very successful, the Ingress spec has fundamental structural limitations. Because the Ingress spec was kept very simple, complex configuration ends up piled into annotations. This makes Ingress manifests non-portable across clusters (e.g. NGINX annotations can’t be understood by Traefik). Additionally, Ingress forces one manifest file to be jointly managed by SysOps (handling domains/SSL) and Developers (handling path routing).

To overcome these limitations, the CNCF developed the Gateway API as the next-generation modern routing standard.

flowchart TD
    Ops["1. Cluster Operator / SysOps (Manages GatewayClass)<br>'Use the Envoy Controller for the Load Balancer'"]
    Ops --> Plat["2. Platform Engineer (Manages Gateway)<br>'Open Port 443 HTTPS with a Wildcard Domain SSL'"]
    Plat --> Dev["3. Application Developer (Manages HTTPRoute)<br>'Map path /payments to Service payment-service'"]

Key Gateway API Advantages #

Separation of Concerns: Separates network management responsibilities declaratively through separate objects (GatewayClass, Gateway, and HTTPRoute).
Built-in Sophistication: Natively supports advanced features like weighted traffic splitting (canary deployments), header-based routing, and URL redirection straight from the API spec without annotation help.
Multi-Tenant Security: Platform Engineers can set strict security policies on the Gateway object to restrict which namespaces may create HTTPRoute objects on specific domains.

Ingress Design Anti-Patterns vs Solutions #

Let’s study some fatal configuration mistakes we often find when implementing Ingress in production clusters.

Anti-Pattern 1: Ignoring the Modern `ingressClassName` Specification #

We deploy an Ingress Resource object without the ingressClassName property, instead relying on the legacy kubernetes.io/ingress.class: "nginx" annotation.

Wrong Manifest Code (Old Style Without an Explicit Class) #

# DON'T USE THIS STYLE ANYMORE
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress-legacy
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx" # LEGACY ANNOTATION: Prone to being ignored by modern clusters
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

The Bad Consequences #

Since Kubernetes version 1.22+, the kubernetes.io/ingress.class annotation has been declared deprecated. If our cluster runs more than one Ingress Controller (e.g. running NGINX for public access and Kong for an internal API Gateway), leaving the ingressClassName property empty can trigger routing failures because no controller feels responsible for processing that manifest.

Solution Code (Modern Declarative Style) #

# SOLUTION: Always use the explicit ingressClassName property
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress-modern
  namespace: production
spec:
  ingressClassName: nginx # Explicitly selects the NGINX Ingress Controller
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

Anti-Pattern 2: Misconfiguring Regex on URL Rewrite Annotations #

We want all requests with the /api URL prefix to be forwarded to our backend web application, but we want the Ingress to strip the /api word when the packet arrives at the backend (e.g. the /api/v1/users query gets rewritten to /v1/users).

Wrong Manifest Code (Rewrite Without a Capturing Group) #

# DON'T DO THIS: Stripping the path crudely
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress-rewrite-failure
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: / # Overwrites the entire path to root!
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api # Without a regex capturing group
            pathType: Prefix
            backend:
              service:
                name: backend-service
                port:
                  number: 80

The Bad Consequences #

With the configuration above, if a client accesses http://app.example.com/api/v1/users, NGINX overwrites the entire URL path to root /. Our backend application receives the request at path / and returns a 404 error because the /v1/users path was completely lost due to the misdirected rewrite process.

Solution Code (Using Regex Capturing Groups Correctly) #

# SOLUTION: Use a regex capturing group to preserve the sub-path
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress-rewrite-success
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2 # Takes the remaining path from the second group ($2)
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api(/|$)(.*) # Group 1: (/|$), Group 2: the remaining path (.*)
            pathType: Prefix
            backend:
              service:
                name: backend-service
                port:
                  number: 80

Practical Ingress Connectivity Debugging Guide #

If our public domain returns a 502 Bad Gateway or 503 Service Temporarily Unavailable error, here are the systematic audit steps:

1. Check the Ingress IP Address #

Make sure the API Server successfully allocated a public external IP to our Ingress object:

kubectl get ingress -n production

If the ADDRESS column is empty after a few minutes, check whether the Ingress Controller is configured with a healthy LoadBalancer Service.

2. Check the Ingress Controller Pod Logs #

The main key to Ingress troubleshooting is in the controller pod’s log files. Find the NGINX Ingress Controller pod in the kube-system namespace or its custom namespace:

# Get logs from the NGINX Ingress Controller
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=100

Review whether there are log lines containing reload errors:

"Configuration is invalid ...": A syntax error occurred in the path regex or SSL secret parameters we defined.
"502 Bad Gateway": The Ingress Controller successfully identified the route, but our backend application Pod isn’t responding (e.g. the Pod crashed or the container port doesn’t match the Service’s targetPort).

Summary #

Ingress minimizes cloud costs: Uses one cloud Load Balancer unit to serve dozens of internal cluster Services through HTTP layer-7 routing.
Understand the Controller vs Resource roles: The Ingress Resource is a static configuration file, while the Ingress Controller (like NGINX) is the active proxy engine executing those rules.
Implement TLS Termination: Free our application Pods’ CPU load from SSL/TLS decryption processes by safely terminating HTTPS connections at the Ingress Controller level.
Use cert-manager for SSL management: Automate the Let’s Encrypt SSL certificate creation and renewal process by integrating cert-manager directly into the Ingress Resource.
Leverage annotations for production features: Configure rate limiting, proxy body size, CORS, and connection timeouts using controller-specific annotations.
Use regex capturing groups for rewrites: Make sure URL rewrite target writing includes capturing groups so sub-path segmentation sent to backend applications isn’t destroyed.

← Previous: DNS & Service Discovery Next: Network Policy →

Ingress #

Ingress Components: Resource vs Controller #

1. The Ingress Resource #

2. The Ingress Controller #

The Ingress Request Handling Flow #

Production Ingress Manifest Structure #

Diving into the pathType Parameter #

TLS Termination and HTTPS Security #

1. Creating the TLS Secret Manually #

2. Automatic Integration with cert-manager (Let’s Encrypt) #

NGINX Ingress Annotations Optimization for Production #

1. Rate Limiting (DDoS & Brute Force Prevention) #

2. Allowing Large File Uploads #

3. CORS (Cross-Origin Resource Sharing) Configuration #

4. Connection Timeout Configuration #

Ingress vs Gateway API: The Future of Network Routing #

Key Gateway API Advantages #

Ingress Design Anti-Patterns vs Solutions #

Anti-Pattern 1: Ignoring the Modern ingressClassName Specification #

Wrong Manifest Code (Old Style Without an Explicit Class) #

The Bad Consequences #

Solution Code (Modern Declarative Style) #

Anti-Pattern 2: Misconfiguring Regex on URL Rewrite Annotations #

Wrong Manifest Code (Rewrite Without a Capturing Group) #

The Bad Consequences #

Solution Code (Using Regex Capturing Groups Correctly) #

Practical Ingress Connectivity Debugging Guide #

1. Check the Ingress IP Address #

2. Check the Ingress Controller Pod Logs #

Summary #

Diving into the `pathType` Parameter #

Anti-Pattern 1: Ignoring the Modern `ingressClassName` Specification #