Rolling Update #

In modern application operations, periodically updating code or configuration is a daily routine. The biggest challenge is how we can launch that new version without disturbing active users at all (zero-downtime deployment). Kubernetes natively solves this challenge using the Rolling Update strategy. Compared to traditional deployment methods that often require a maintenance window in the middle of the night, Rolling Update in Kubernetes works by gradually replacing old Pods with new Pods.

However, achieving true zero-downtime isn’t as simple as letting the default configuration run. We must mathematically configure failure tolerance, align graceful shutdown lifecycles, and design accurate health probes (readiness probes). This article dissects the internal Rolling Update mechanism, transition parameter calculations, network routing synchronization, and a troubleshooting guide when the update process gets stuck.

How the Deployment Controller Works Internally #

When we update the Pod spec (like changing the container image version tag) on a Deployment object, Kubernetes doesn’t directly overwrite the running Pods. The Deployment Controller acts as a high-level manager coordinating Pod creation and deletion through ReplicaSet objects.

Here are the state transition stages managed by the Deployment Controller:

  1. New ReplicaSet Creation: The Deployment Controller creates a second ReplicaSet (let’s call it v2) alongside the currently active ReplicaSet (v1).
  2. Gradual Scaling Up (Scaling Up v2): The v2 ReplicaSet starts creating new Pods with v2 code versions.
  3. Health Verification: v2 Pods must pass the Readiness Probe check. Before this probe returns a success status, v2 Pods are considered not ready to serve traffic and won’t be added to the network endpoint list.
  4. Gradual Scaling Down (Scaling Down v1): After one or several v2 Pods are confirmed healthy, the Deployment Controller instructs the v1 ReplicaSet to slowly kill some v1 Pods.
  5. Cycle Repetition: This process repeats until all v1 Pods are deleted (scale zero) and the v2 ReplicaSet fully holds the entire desired replica count.
sequenceDiagram
    participant DC as Deployment Controller
    participant RS1 as ReplicaSet v1 (Old)
    participant RS2 as ReplicaSet v2 (New)
    participant Pod2 as Pod v2 (Candidate)
    participant SVC as EndpointSlice / Service
    
    DC->>RS2: Order Scale Up (+1 v2 Pod)
    RS2->>Pod2: Create v2 Pod
    Note over Pod2: Container Initialization & Probe Execution
    Pod2-->>DC: Readiness Probe Success (200 OK)
    DC->>SVC: Add v2 Pod IP to the Routing List
    DC->>RS1: Order Scale Down (-1 v1 Pod)
    Note over RS1: Gracefully delete 1 v1 Pod
    Note over DC: Repeat the cycle until v1 = 0

Parameter Calculations: maxSurge and maxUnavailable #

The speed, extra resource usage, and failure tolerance level during the Rolling Update process are strictly controlled by two parameters under the rollingUpdate property:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0

We can set both parameters’ values using absolute numbers (e.g. 1 or 2) or percentages of the desired replica total (e.g. 25%).

1. maxSurge #

Determines the maximum number of extra Pods allowed above the desired replica capacity during the update process.

  • Formula: $\text{Maximum Pod Count} = \text{Desired Replicas} + \text{maxSurge}$
  • Implication: A high maxSurge value speeds up the deployment process because Kubernetes can create many new Pods in parallel. However, this requires sufficient leftover CPU/Memory capacity (headroom) on the cluster nodes.

2. maxUnavailable #

Determines the maximum number of Pods allowed to be unavailable (dead or not ready) during the update process.

  • Formula: $\text{Minimum Available Pod Count} = \text{Desired Replicas} - \text{maxUnavailable}$
  • Implication: If we set maxUnavailable: 0, Kubernetes guarantees our service capacity never drops below 100% of the total replicas we requested during the transition.
  • Configuration: maxSurge: 1 (or 25%), maxUnavailable: 0
  • Characteristics: Guarantees full 100% service capacity at all times. Old Pods are never killed before new Pods are confirmed healthy by the readiness probe.
  • Trade-off: Deployments run slower because they must wait for new Pods to be ready one by one before continuing the next cycle.

Pattern B: Resource-Efficient (Limited Clusters) #

  • Configuration: maxSurge: 0, maxUnavailable: 1 (or 25%)
  • Characteristics: Needs no extra cluster resources at all. Kubernetes immediately kills one old Pod to free a scheduling slot for a new Pod.
  • Trade-off: During transition, our service capacity drops by one replica. Not suitable for applications already near 100% utilization under peak load.

Pattern C: Fast Large-Scale Updates #

  • Configuration: maxSurge: 50%, maxUnavailable: 0
  • Characteristics: Very fast. Half of the new Pods are created instantly at the start of the process.
  • Trade-off: Requires very large node capacity headroom in the cluster.

The Vital Role of Probes in Kubernetes Networking #

A safe Rolling Update mechanism heavily depends on the accuracy of the Readiness Probe. When a v2 Pod is newly created, the Kubelet on the worker node triggers the container startup. If the container reaches Running status, that only indicates the main process (like node server.js or java -jar) has been executed. That process usually needs a few seconds to load libraries, open database connections, or warm up caches.

If we don’t define a readinessProbe:

  1. Kubernetes immediately considers that v2 Pod healthy (Ready).
  2. The Service Controller immediately adds the v2 Pod IP to the traffic routing list.
  3. Kube-proxy and the Ingress Controller start sending real user traffic to the v2 Pod that isn’t fully ready. Users get HTTP 502 Bad Gateway or 503 Service Unavailable errors.
  4. Simultaneously, Kubernetes immediately kills v1 Pods because it considers the transition task complete.
flowchart TD
    subgraph NoProbe["WITHOUT PROBE"]
        Run1["v2 Pod Running"] --> Ready1["Immediately Marked Ready"] --> Traffic1["Incoming Traffic"] --> Crash1["CRASH/ERROR 502"]
        Crash1 --> Kill1["Killed Instantly"] --> PodV1_1["Old v1 Pod"]
    end

    subgraph WithProbe["WITH PROBE"]
        Run2["v2 Pod Running"] --> Wait2["Wait for Probe Success / 200 OK"] --> Ready2["Marked Ready"] --> Traffic2["Incoming Traffic (Smooth)"]
        Traffic2 --> Kill2["Killed Gradually"] --> PodV1_2["Old v1 Pod"]
    end

The Importance of minReadySeconds #

Besides probes, the minReadySeconds property is very important to add. This property instructs Kubernetes to wait N seconds after a new Pod is marked Ready before proceeding to the next step.

If our application has a hidden memory bug causing it to crash 5 seconds after receiving real traffic, minReadySeconds: 30 saves us. Kubernetes stops the update process because the new Pod dies before the stabilization time limit passes, so the remaining v1 Pods safely keep serving.

The Graceful Shutdown Cycle #

When the Deployment Controller decides to kill old Pods (v1), the connection termination process must be managed smoothly so user requests currently running inside the container aren’t cut off mid-way.

The Pod Shutdown Flow in Kubernetes: #

  1. Pod Changed to Terminating Status: Kubernetes removes the Pod IP from the Service Endpoint list. Ingress and kube-proxy start updating their routing tables so they don’t send new connections to that Pod.
  2. preStop Hook Execution: If we configure a preStop lifecycle hook on the container, Kubernetes runs that command first.
  3. SIGTERM Signal Sent: Kubernetes sends the SIGTERM signal to the main process (PID 1) inside the container. The application must catch this signal, stop accepting new connections, finish the in-flight request queue, and close database connections cleanly.
  4. Grace Period Timeout: Kubernetes waits until the terminationGracePeriodSeconds time limit (default 30 seconds) passes.
  5. SIGKILL Signal Sent: If the application hasn’t died after the time limit passes, the host kernel sends SIGKILL to force-kill the process.
flowchart TD
    Start["Deployment Controller triggers Scale Down"] --> Step1["Pod status changed to 'Terminating'"]
    Step1 --> Step2["Pod IP removed from the Service EndpointSlice"]
    
    subgraph Parallel["Running in Parallel"]
        Step3["Kube-proxy & Ingress update routing tables (1-5 seconds)"]
        Step4["Kubelet triggers the 'preStop' hook on the container"]
    end
    
    Step2 --> Step3
    Step2 --> Step4
    
    Step3 --> Step5["Kubelet sends SIGTERM to PID 1"]
    Step4 --> Step5
    
    Step5 --> Step6{"Does the process die<br>before the Grace Period ends?"}
    
    Step6 -- "Yes" --> EndSafe["Pod Cleanly Deleted"]
    Step6 -- "No" --> Step7["Kubelet force-sends SIGKILL"]
    
    Step7 --> EndForced["Pod Force-Deleted"]
    
    style EndSafe stroke:#388e3c,stroke-width:2px
    style EndForced stroke:#d32f2f,stroke-width:2px

[!IMPORTANT] Why is a preStop hook with sleep often needed? The routing table synchronization process across worker nodes by kube-proxy takes a few seconds (propagation delay). If the application immediately responds to SIGTERM by closing connection sockets instantly, new user connections sent during that propagation gap fail. Applying a preStop hook with a sleep 5 command gives the network routing tables time to fully update before the application starts shutting down its services.

Anti-Patterns vs Best Solutions #

Let’s study some Rolling Update configuration mistakes in production environments along with their best fixes.

Anti-Pattern 1: Setting maxUnavailable to 100% on Production Clusters #

Setting the maxUnavailable parameter too large (or equal to the replica count) to speed up releases. This action destroys the zero-downtime foundation because Kubernetes is allowed to kill all old Pods at once before new Pods are ready.

# ✗ ANTI-PATTERN: Allowing all Pods to die simultaneously during updates
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 100% # ← CRITICAL: All v1 Pods killed instantly, total downtime occurs

Best Solution #

Use strict limits with maxUnavailable: 0 so the minimum service capacity stays fully maintained.

# ✓ SOLUTION: Safe configuration guaranteeing Pod availability
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0 # ← Safe, old Pods are only killed after new Pods are ready to receive traffic

Anti-Pattern 2: Not Configuring a preStop Hook for High-Traffic HTTP Web Applications #

During rolling updates, users sometimes periodically get HTTP 502 Bad Gateway errors (flapping). This happens because the container immediately processes shutdown upon receiving SIGTERM, while the Ingress Controller is still routing remaining requests to that Pod due to internal network synchronization delays.

Best Solution #

Add a preStop lifecycle hook that does a brief delay before the Kubelet sends SIGTERM to the application.

# ✓ SOLUTION: Adding a preStop hook to mitigate propagation delay
spec:
  containers:
  - name: web-app
    image: company/web-app:v2.1.0
    lifecycle:
      preStop:
        exec:
          # Gives kube-proxy / Ingress time to remove the Pod IP from routing
          command: ["/bin/sh", "-c", "sleep 10"]
    terminationGracePeriodSeconds: 45 # Give enough time (longer than the preStop sleep)

Best Production Deployment Manifest #

Here’s a production-ready Deployment manifest example applying a safe Rolling Update strategy, complete with availability controls, health probes, and graceful shutdown:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
  namespace: finance
  labels:
    app: billing-service
spec:
  replicas: 4
  # Sets the stability wait tolerance for new Pods
  minReadySeconds: 15
  # Overall deployment timeout limit (300 seconds)
  progressDeadlineSeconds: 300
  # Number of ReplicaSet histories stored for instant rollback
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1             # Maximum 1 extra Pod above 4 replicas (total 5)
      maxUnavailable: 0       # Always ensure at least 4 Pods are ready to receive traffic
  selector:
    matchLabels:
      app: billing-service
  template:
    metadata:
      labels:
        app: billing-service
    spec:
      terminationGracePeriodSeconds: 60 # Matches the internal application cleanup duration
      containers:
      - name: billing-app
        image: company/billing-app:v2.1.0
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "1000m"
            memory: "512Mi"
        lifecycle:
          preStop:
            exec:
              # Pause so traffic isn't routed to this dying Pod
              command: ["/bin/sh", "-c", "sleep 15"]
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
          failureThreshold: 3

CLI Command Guide for Rollout Management #

We can interactively control and monitor the Rolling Update process using the kubectl CLI:

1. Monitoring Deployment Progress #

To see the real-time current status of the Pod update:

kubectl rollout status deployment/billing-service -n finance
# Success output: deployment "billing-service" successfully rolled out

2. Checking Revision History #

Every time a Pod spec change is saved, Kubernetes records it as a new revision:

kubectl rollout history deployment/billing-service -n finance

To see the spec details of a specific revision:

kubectl rollout history deployment/billing-service --revision=2 -n finance

3. Pausing and Resuming Releases (Pause/Resume) #

Very useful if we want to do gradual modifications or a quick investigation in the middle of a rollout process:

# Pause the running rollout
kubectl rollout pause deployment/billing-service -n finance

# Resume the rollout after the investigation is done
kubectl rollout resume deployment/billing-service -n finance

4. Canceling Releases (Rollback) #

If a system failure is detected on the new version, we can immediately cancel it and return to the previous version:

# Rollback to the revision right before the current one
kubectl rollout undo deployment/billing-service -n finance

# Rollback to a specific revision based on the history number
kubectl rollout undo deployment/billing-service --to-revision=2 -n finance

Common Problem Diagnosis (Troubleshooting) #

If our Rolling Update process stops mid-way (stuck), run the following systematic diagnosis steps:

  1. Check the Deployment Status:

    kubectl describe deployment billing-service -n finance

    Pay attention to the Conditions section. If there’s a ProgressDeadlineExceeded message, the progressDeadlineSeconds time limit was exceeded because a container failed to start.

  2. Identify the Involved ReplicaSets:

    kubectl get replicaset -n finance -l app=billing-service

    Compare the DESIRED, CURRENT, and READY columns. If the new ReplicaSet has a high DESIRED count but READY is zero, the main problem is in the new version Pods.

  3. Find the Problematic Pods:

    kubectl get pods -n finance -l app=billing-service

    Look for Pods with non-Running status or with a READY column value of 0/1.

  4. Investigate Logs and Pod Details:

    # Check container logs for application stacktrace errors
kubectl logs <new-pod-name> -n finance

# Check system events for probe failures, resource shortages, or image pulls
kubectl describe pod <new-pod-name> -n finance

The Most Common Root Causes: #

  • ImagePullBackOff: The image tag is mistyped, or the worker node lacks authorization credentials to access the private container registry repository.
  • CrashLoopBackOff: The application crashes right after initial initialization due to missing required environment variables (ConfigMap/Secret).
  • Node Resource Shortage (Pending status): The cluster has no worker node with free CPU/Memory capacity to host the new extra (maxSurge) Pods.

Summary #

  • Understand maxSurge & maxUnavailable — Use the maxSurge: 25% and maxUnavailable: 0 combination as the standard production configuration to guarantee full 100% capacity availability during releases.
  • Readiness Probes must be installed — Without probes, Kubernetes routes user traffic to containers that haven’t finished initial initialization.
  • Use minReadySeconds as a buffer — The extra wait time after Pods become ready helps detect hidden startup failures before all old Pods are killed.
  • Use preStop hooks for graceful shutdown — A short sleep command in the preStop hook gives the kube-proxy network time to sync routing tables before the container dies.
  • Maintain revision history limits — Set the revisionHistoryLimit property to limit stale ReplicaSet counts in the cluster while still keeping backups for fast rollbacks.
  • Monitor status regularly — Leverage the kubectl rollout status command in our CI/CD pipeline to validate deployment success before marking the release complete.

← Previous: Deployment Strategy Overview   Next: Blue/Green Deployment →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact