High Availability #

For many organizations, deploying applications to Kubernetes with the replicas: 2 parameter in Deployment manifests is often considered enough to claim the application has High Availability (HA). However, in reality, this is a wrong understanding. If the scheduler places both replica pods on the same physical Node, then a hardware failure on that Node instantly kills our entire service completely. Real High Availability requires intentional workload distribution strategies. We must ensure Pods are spread across several different Nodes, Nodes are spread across several Availability Zones (AZs), and the cluster has automatic defenses against scheduled maintenance operations like node OS upgrades. This article discusses advanced techniques for securing our application availability from partial and total infrastructure failures.

Why Two Replicas Alone Aren’t Enough #

By default, the Kubernetes scheduling algorithm (kube-scheduler) prioritizes Node resource usage efficiency. If a Node has loose memory capacity and is nearby, the scheduler tends to place all new replica pods on that Node.

DEPLOYMENT SCENARIO WITH 2 REPLICAS:

  1. Without Distribution Rules (Default Scheduler):
     Worker Node 1: [payment-api-pod-1] [payment-api-pod-2]  <-- Both stacked on one node!
     Worker Node 2: (Empty / Other workloads)
     Worker Node 3: (Empty)
     
     Result: Worker Node 1 suffers a power outage -> Both Pods die simultaneously -> DOWNTIME.

  2. With Anti-Affinity Rules:
     Worker Node 1: [payment-api-pod-1]
     Worker Node 2: [payment-api-pod-2]                     <-- Evenly spread!
     Worker Node 3: (Empty)
     
     Result: Worker Node 1 dies -> [payment-api-pod-1] is lost, but [payment-api-pod-2] on Node 2 
             stays active serving user transactions -> ZERO DOWNTIME.

To guarantee high availability at the production level, we must declaratively direct the scheduler using the Pod Anti-Affinity and Topology Spread Constraints features.

Arranging Pod Distribution via Pod Anti-Affinity #

Pod Anti-Affinity lets us prohibit Kubernetes from placing multiple Pods with identical labels on the same topology domain (like Nodes or Zones).

Anti-affinity rules are divided into two strictness levels:

Strict Rules (requiredDuringSchedulingIgnoredDuringExecution): Hard rules that must be met. If the scheduler can’t find qualifying Nodes (e.g. we request 4 replicas spread across different Nodes, but our cluster only has 3 Nodes), the 4th Pod is left stuck in the Pending status forever.
Soft Rules (preferredDuringSchedulingIgnoredDuringExecution): Priority rules. The scheduler tries spreading Pods if possible, but if Node capacity is insufficient, the scheduler still places Pods on the same Node instead of letting them fail to run.

# File: k8s/production-affinity.yaml
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      affinity:
        podAntiAffinity:
          # 1. HARD RULE (Must be on different physical Nodes)
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: payment-api
            topologyKey: "kubernetes.io/hostname" # "hostname" represents a single Node domain
          
          # 2. PRIORITY RULE (Preferably in different Availability Zones if possible)
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100 # The highest priority value (1-100)
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: payment-api
              topologyKey: "topology.kubernetes.io/zone" # represents the Availability Zone (AZ) domain

Main Topology Domains (`topologyKey`) #

topologyKey determines the geographic or infrastructure boundary where anti-affinity is applied:

kubernetes.io/hostname: Restricts so only a maximum of 1 Pod is allowed per physical Node.
topology.kubernetes.io/zone: Restricts Pods from stacking in the same Availability Zone (e.g. in AWS: ap-southeast-1a vs ap-southeast-1b).
topology.kubernetes.io/region: Regional level (rarely used for single clusters, usually for large multi-region clusters).

Topology Spread Constraints: Precise Cross-Zone Scaling #

Although Pod Anti-Affinity is very reliable for preventing Pod stacking, it has a limitation: it can’t control the balance of the spread Pod count. For example, if we have 10 Pod replicas and 3 Availability Zones (AZ-A, AZ-B, AZ-C), AZ-level anti-affinity can’t prevent the scheduler from placing 8 Pods in AZ-A, 1 Pod in AZ-B, and 1 Pod in AZ-C (because each AZ still has running Pods, so anti-affinity rules aren’t violated).

This imbalance is very dangerous. If AZ-A suffers a total power disruption, we instantly lose 80% of our application’s compute capacity.

Topology Spread Constraints (TSC) elegantly solves this problem by letting us specify the maximum imbalance tolerance limit (called maxSkew) between topology domains.

Understanding maxSkew #

maxSkew is the maximum allowed difference in active Pod counts between the zone with the most Pods and the zone with the fewest Pods in the target topology domain.

$$\text{Actual Skew} = \text{Number of Pods in the Densest Zone} - \text{Number of Pods in the Sparseest Zone}$$ $$\text{Validation Requirement: } \text{Actual Skew} \le \text{maxSkew}$$

CASE: 9 POD REPLICAS IN 3 AVAILABILITY ZONES (AZ-A, AZ-B, AZ-C)

  1. Without TSC (High & Risky Skew):
     AZ-A: [Pod1] [Pod2] [Pod3] [Pod4] [Pod5] [Pod6] (6 Pods)
     AZ-B: [Pod7] [Pod8]                             (2 Pods)
     AZ-C: [Pod9]                                    (1 Pod)
     
     Actual Skew = 6 (densest) - 1 (sparsest) = 5.
     Danger: If AZ-A goes down, 67% of the application capacity is lost at once.

  2. With TSC (maxSkew: 1):
     AZ-A: [Pod1] [Pod2] [Pod3]                      (3 Pods)
     AZ-B: [Pod4] [Pod5] [Pod6]                      (3 Pods)
     AZ-C: [Pod7] [Pod8] [Pod9]                      (3 Pods)
     
     Actual Skew = 3 - 3 = 0 (meets the <= 1 requirement).
     Result: Perfect distribution. If one AZ goes down, we only lose 33% of capacity.

Production-Level TSC Manifest #

# File: k8s/production-tsc.yaml
spec:
  replicas: 6
  template:
    metadata:
      labels:
        app: core-api
    spec:
      topologySpreadConstraints:
      # 1. Limit imbalance across Availability Zones (Strict)
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule # Don't run new Pods if they create skew > 1
        labelSelector:
          matchLabels:
            app: core-api
            
      # 2. Limit imbalance across Host Nodes (Soft / Flexible)
      - maxSkew: 2
        topologyKey: "kubernetes.io/hostname"
        whenUnsatisfiable: ScheduleAnyway # Still run Pods even if skew > 2 when Nodes are full
        labelSelector:
          matchLabels:
            app: core-api

Pod Disruption Budgets (PDBs): Protection During Node Maintenance #

Cluster high availability isn’t only threatened by unexpected hardware failures (Involuntary Disruptions like power outages or Node disk damage). High availability is actually more often disturbed by intentional cluster maintenance actions (Voluntary Disruptions) like:

Node draining by administrators for host kernel OS version upgrades.
Automatic Node removal by the Cluster Autoscaler during cost efficiency actions (scale-down).
Manual Pod eviction processes.

To protect critical application Pods from mass outages during these maintenance processes, we must define Pod Disruption Budgets (PDBs). PDBs act as traffic police prohibiting the API Server from killing Pods if the remaining active Pod count would drop below the minimum safety threshold.

# File: k8s/production-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
  namespace: prod-apps
spec:
  # Option 1: Specify the minimum Pod count that must always be healthy & active
  minAvailable: 2 
  
  # Option 2 (Alternative): Specify the maximum Pod count allowed to die simultaneously
  # maxUnavailable: 1 # Or use a percentage, e.g. "25%"
  
  selector:
    matchLabels:
      app: payment-api

PDB Eviction Blocking Simulation #

When administrators run Node drain commands for OS upgrade purposes:

# The administrator drains worker-1 node for maintenance
$ kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data

The Kubernetes API Server checks all registered PDBs in the cluster. If evicting payment-api-pod-1 on node-worker-1 would cause the remaining active pod count in the cluster to drop below minAvailable (2 pods), the drain command is postponed (blocked).

The administrator terminal displays the delay status visually:

evicting pod prod-apps/payment-api-pod-1
error when evicting pod "payment-api-pod-1" (retrying after 5s): 
Cannot evict pod as it would violate the pod's disruption budget.

The Kubelet waits until the replacement pod scheduled on another Node (e.g. node-worker-2) successfully starts and passes the readiness probe before finally allowing the old pod’s removal on node-worker-1.

Graceful Shutdown & Zero-Downtime Endpoints #

Deploying Pods across many Nodes and Zones doesn’t guarantee zero downtime if our application doesn’t properly handle the container termination process (graceful shutdown).

When Kubernetes decides to stop a Pod (e.g. during rolling updates or scale-downs), two asynchronous process flows run in parallel inside the cluster:

flowchart TD
    DeleteRequest["1. API Server Receives the Pod Delete Command"] --> ParallelNodes{"2. Parallel Async Flows"}
    
    ParallelNodes -->|"Kubelet Node Flow"| KubeletSignal["3. Kubelet Sends the SIGTERM Signal to the Container"]
    KubeletSignal --> RunPreStop["4. Runs the preStop Lifecycle Hook Cycle"]
    RunPreStop --> WaitGrace["5. Waits terminationGracePeriodSeconds"]
    WaitGrace --> ForceKill["6. Sends the SIGKILL Signal (Force Kill the Container)"]
    
    ParallelNodes -->|"Network Control Plane Flow"| ServiceController["7. The Endpoint Controller Removes the Pod IP from Endpoints"]
    ServiceController --> KubeProxy["8. kube-proxy Updates iptables/IPVS Rules on All Nodes"]
    KubeProxy --> IngressRoute["9. The Ingress Controller Stops Sending New Traffic to the Pod"]

The Network Race Condition Problem #

Because the flows above run asynchronously, time gaps (latency) often happen. The Kubelet on the Node where the Pod runs may receive the SIGTERM signal and instantly stop our application process in milliseconds. However, the iptables/IPVS rule update process by kube-proxy on other Nodes across the cluster takes up to 2-3 seconds to propagate.

As a result, during that 2-3 second gap, the Ingress Controller or other Pods still send new HTTP traffic packets to the already-dead Pod IP address, producing HTTP 502 Bad Gateway network transaction errors for end users.

Solution: Applying preStop Hooks and Robust Probes #

To mitigate the network time gap above, we must insert a sleep delay using the preStop lifecycle hook before the container is allowed to process the SIGTERM shutdown signal.

# File: k8s/production-graceful.yaml
spec:
  template:
    spec:
      # Give sufficient transition time for the application to finish active transactions
      terminationGracePeriodSeconds: 60 # Default is only 30 seconds
      containers:
      - name: api-app
        image: registry.company.com/apps/api:v1.0.0
        
        lifecycle:
          preStop:
            exec:
              command: 
              - "/bin/sh"
              - "-c"
              # A 10-second sleep delay holds the app from immediately processing SIGTERM.
              # Gives time for kube-proxy to remove the Pod IP from all cluster iptables tables.
              - "sleep 10"
              
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 2 # Quickly detect if the pod suffers disruptions
          
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 15

With this configuration, when the delete command arrives:

The container is ordered to sleep for 10 seconds via preStop. During this period, the container stays active and can process remaining old transactions.
While the container sleeps, the Endpoint Controller removes the Pod IP from the active Endpoints list.
The Ingress Controller stops routing new traffic to that Pod because its name was removed from the active route list.
After 10 seconds pass, the container wakes up and receives the SIGTERM signal, then cleanly finishes internal processes (graceful exit) without any risk of dropping connections mid-way.

Control Plane High Availability (HA Multi-Master) #

Our application’s high availability means nothing if our cluster’s Control Plane itself dies. At the production level, we must configure the HA Multi-Master architecture:

flowchart TD
    LB["EXTERNAL LOAD BALANCER (API Server Access via Port 6443)"]
    CP1["Control Plane 1 (API Server)"]
    CP2["Control Plane 2 (API Server)"]
    CP3["Control Plane 3 (API Server)"]
    ETCD["etcd CLUSTER (3 or 5 Odd Nodes for Raft Quorum Consensus)"]

    LB --> CP1
    LB --> CP2
    LB --> CP3

    CP1 --> ETCD
    CP2 --> ETCD
    CP3 --> ETCD

External Load Balancer: Place a high-level TCP Load Balancer in front of all API Server master nodes to route kubectl traffic and Kubelet connections from worker nodes if one API Server dies.
etcd Quorum: The etcd cluster consistency database uses the Raft consensus algorithm requiring an odd number of nodes to avoid Split-Brain situations.
- 3 etcd Nodes: Allows a maximum of 1 dead node without losing quorum.
- 5 etcd Nodes: Allows a maximum of 2 dead nodes simultaneously.