Recreate #

In the world of cloud-native software engineering, the concept of downtime (service interruption time) is something to be avoided whenever possible. We race to configure Rolling Update, Blue/Green, or Canary strategies to guarantee 100% service availability (high SLA) during new version launches. However, there are times when forcing a zero-downtime release is actually the most dangerous architectural decision. This condition usually happens when the system undergoes fundamental, non-backward-compatible changes (breaking changes), whether at the database level, message formats, or exclusive resource locking mechanisms.

The Recreate strategy is a release approach where Kubernetes stops (scales down to zero) all old version (v1) Pod replicas simultaneously, waits until all cleanup processes finish, and only then runs the new version (v2) ReplicaSet fully. Although this strategy inherently creates a downtime window, there are specific production scenarios where the Recreate strategy isn’t just acceptable — it’s the only choice guaranteeing data integrity and cluster stability.

The Recreate Strategy’s Architectural Mechanism #

When we update a Deployment manifest using the Recreate strategy, the Pod transition lifecycle is strictly managed by the Deployment Controller through the following structured steps:

[ V1 REVISION ] ──────────────► [ V1 TERMINATION ] ──────────────► [ DOWNTIME WINDOW ] ──────────────► [ V2 STARTUP ] ──────────────► [ V2 REVISION ]
Pod 1: Running (v1)            Pod 1: Terminating (v1)         (No Running Pods)                   Pod 1: Pending/Init (v2)       Pod 1: Running (v2)
Pod 2: Running (v1)            Pod 2: Terminating (v1)                                             Pod 2: Pending/Init (v2)       Pod 2: Running (v2)
  1. Mass v1 Termination: The Deployment Controller scales the v1 ReplicaSet down to zero. All v1 Pods simultaneously enter Terminating status.
  2. Graceful Shutdown: v1 Pods are given a tolerance time (per the terminationGracePeriodSeconds property) to finish active HTTP requests, flush memory queues, and release database connections.
  3. Exclusive Resource Release: After v1 Pods are truly deleted from the system, Kubernetes releases all resource references locked by v1 Pods, like Persistent Volumes (PVs) or static port bindings.
  4. Downtime Window: In this phase, no Pods are operating to serve user traffic. The production Service endpoints are empty, and incoming traffic gets HTTP 503 Service Unavailable responses.
  5. v2 Initialization: The Deployment Controller detects all v1 Pods are cleanly gone from the cluster, then starts instructing the v2 ReplicaSet to create and run all v2 Pods in parallel.
  6. Startup & Probes: v2 Pods go through the application bootstrap process, run readinessProbes, and after being confirmed healthy, the Service starts flowing traffic back to the new version.

Absolute Use Case Scenarios for the Recreate Strategy #

Understanding when we must enable the Recreate strategy is the key to maintaining data integrity. Here are four main use cases where Recreate is the best choice:

1. Radical Database Schema Changes (Breaking Schema Migrations) #

If we apply database schema migrations that cut backward compatibility (non-backward compatible), running zero-downtime strategies (like Rolling Update) is very dangerous.

Let’s say we change a database table column name from user_name to username:

  • If Using Rolling Update: Old v1 Pods and new v2 Pods run side by side for a few minutes. As soon as the database schema migration executes at the start of the release, the user_name column gets changed to username. As a result, all still-active v1 Pods serving users crash instantly because their database queries (SELECT user_name FROM users) fail to find that column.
  • If Using Recreate: We guarantee all v1 Pods needing the old column are completely dead before the database migration script executes and v2 Pods run.

2. Data Serialization Inconsistencies (Message Queue Serialization Shift) #

This condition happens when the data exchange format between services fundamentally changes, e.g. migrating from JSON format to Google Protocol Buffers (Protobuf) on message queues.

  • If Using Rolling Update: v2 producers start publishing Protobuf-format messages into the message broker (like RabbitMQ or Kafka). Still-running v1 consumers pull those messages and suffer fatal failures (crash loops) because they can’t parse the Protobuf format.
  • If Using Recreate: We shut down all v1 producers and consumers simultaneously, drain the message queue, and only then turn on v2 producers and consumers in unison.

3. Singleton Workloads & Exclusive Resource Locking #

Some applications are designed as Singletons (only exactly one instance may run across the entire cluster).

  • Examples: An internal scheduler periodically pulling data from third-party APIs, or a stateful application attaching a cloud-based storage disk (Persistent Volume) with ReadWriteOnce (RWO) access mode.
  • Rolling Update Consequences: If we try running a second instance to replace the first instance, the cloud provider host rejects the disk volume attachment because the volume is still exclusively bound to the first instance’s host node. This triggers a Multi-Attach error deadlock. The Recreate strategy cleanly releases the volume lock first before attaching it to the new container.
sequenceDiagram
    participant DC as Deployment Controller
    participant Node1 as Worker Node 1
    participant Disk as Persistent Volume (RWO)
    participant Node2 as Worker Node 2
    
    DC->>Node1: Delete the v1 Pod (Scale Down to 0)
    Node1->>Disk: Release the Volume lock (Detach Disk)
    Disk-->>DC: Volume Successfully Released (Available)
    Note over DC: Downtime Phase Done
    DC->>Node2: Create the v2 Pod (Scale Up to 1)
    Node2->>Disk: Lock the Volume to the New Node (Attach Disk)
    Note over Node2: v2 Pod Ready to serve queries

Downtime Window Minimization Tactics #

Although we accept the fact that Recreate creates downtime, we have an operational responsibility to shorten that downtime duration as much as possible (often targeted below 30 seconds). Here are optimization tactics we can apply:

1. Image Cache Warming (Pre-pull Image / Image Prefetching) #

Most of the downtime duration in the Recreate strategy is spent by worker nodes downloading the new container image from the external registry into the node’s local storage. If the image size reaches gigabytes, downtime can take minutes.

Solution #

Before triggering the Deployment update, run a temporary Kubernetes Job or standby DaemonSet with the single task of downloading the new image to all worker nodes in the cluster. When the Recreate process executes, worker nodes instantly run the new container from the local cache.

# ✓ SOLUTION: Prefetch DaemonSet to download the image to all nodes before release
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-prefetcher
  namespace: tools
spec:
  selector:
    matchLabels:
      name: prefetcher
  template:
    metadata:
      labels:
        name: prefetcher
    spec:
      # This container dies right after the image is downloaded to the host node
      initContainers:
      - name: puller
        image: company/my-large-app:v2.0.0-new # The new image to be deployed
        command: ["echo", "Image pulled successfully"]
      containers:
      - name: sleep
        image: alpine:3.18
        command: ["sleep", "10"]

2. Optimizing terminationGracePeriodSeconds #

By default, Kubernetes waits up to 30 seconds to give shutdown time before force-killing containers. If our application can actually close all its connections cleanly within 5 seconds, keeping the default 30-second value needlessly extends the downtime period.

Solution #

Measure our application’s shutdown time in the staging environment, then precisely adjust the terminationGracePeriodSeconds value.

spec:
  # Reduce the shutdown wait period if the app can safely die quickly
  terminationGracePeriodSeconds: 10

3. Diverting Traffic to a Maintenance Page #

During downtime, users must not see raw browser error pages (like HTTP 503 or Connection Refused). We must dynamically divert traffic to a user-friendly maintenance page.

Solution #

Use Ingress Controller annotation features to detect the absence of healthy Pods (empty endpoints) and automatically divert traffic to a backup static backend.

# ✓ SOLUTION: Ingress configuration with a Custom Error Page during Recreate Downtime
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/custom-http-errors: "502,503"
    # Routes HTTP 502/503 errors to the static maintenance page service
    nginx.ingress.kubernetes.io/default-backend: maintenance-page-service
spec:
  ingressClassName: nginx
  rules:
  - host: app.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-core-service # The main application Service being Recreated
            port:
              number: 80

Anti-Patterns vs Best Solutions #

Let’s study some critical mistakes when operating the Recreate strategy along with their best fixes.

Anti-Pattern 1: Ignoring Image Pre-pull on Production Recreate Releases #

Doing a Deployment update with the Recreate strategy without ensuring the new image is available in worker node caches. The Kubelet deletes all old Pods first, then starts downloading the new image. If the cluster’s internet connection is disrupted or the image size is very large, cluster downtime drags on for tens of minutes.

Best Solution #

Use the pre-pulling tactic (DaemonSet prefetcher) or make sure our CI/CD pipeline runs an image cache readiness verification on nodes before triggering the image tag change on the Deployment.

Anti-Pattern 2: Setting Too-Short Grace Periods for Financial Transactions #

Shortening the terminationGracePeriodSeconds parameter to a very small value (e.g. 2 seconds) on a financial payment processing microservice to minimize downtime. When the Recreate process starts, in-flight payment transactions get force-cut by the SIGKILL signal. This triggers transaction data inconsistencies (data corruption) requiring complicated manual reconciliation.

Best Solution #

Don’t sacrifice transaction safety for downtime. If transactions need a maximum of 20 seconds to finish, set a grace period of at least 25 seconds, and run the deployment during lowest traffic hours (off-peak hours).

Complete Production Recreate Deployment Manifest #

Here’s a production-ready Deployment manifest example safely using the Recreate strategy, complete with resource limits, health probes, and orderly shutdown handling:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-billing-engine
  namespace: finance
  labels:
    app: billing-engine
spec:
  # This app is a singleton (only 1 replica may run)
  replicas: 1
  # Enables the Recreate strategy to cleanly release exclusive volumes
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: billing-engine
  template:
    metadata:
      labels:
        app: billing-engine
    spec:
      # Optimizes the shutdown wait period per the application's database cleanup characteristics
      terminationGracePeriodSeconds: 15
      containers:
      - name: engine
        image: company/billing-engine:v2.5.0
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          periodSeconds: 10
        volumeMounts:
        - name: transaction-logs
          mountPath: /var/log/billing
      volumes:
      - name: transaction-logs
        persistentVolumeClaim:
          claimName: billing-log-pvc # PVC with ReadWriteOnce access mode

Recreate Release Audit Checklist #

Before triggering a deployment using the Recreate strategy, make sure our engineering team audits using the following readiness checklist:

DOWNTIME NEEDS ANALYSIS:
  □ The Recreate strategy usage has been approved by the business and operations teams.
  □ The release is scheduled during the lowest traffic hours (off-peak hours).
  □ A maintenance announcement page has been prepared at the Ingress level.

SPEED & IMAGE STRATEGY OPTIMIZATION:
  □ The new container image has been cached to all worker nodes via a DaemonSet prefetcher.
  □ The 'terminationGracePeriodSeconds' parameter is set in balance (not too long, not cutting transactions).
  □ The application's initial startup time (startup bootstrap) is optimized to run as fast as possible.

RESOURCE & VOLUME LOCKING VERIFICATION:
  □ All RWO disk volumes are confirmed to auto-detach from old host nodes.
  □ Old database connections are gracefully cut before the new database migration script executes.
  □ A fast rollback procedure (rollout undo) is prepared if the new version fails to function.

Summary #

  • Choose Recreate for non-backward-compatible databases — If the old application version crashes immediately due to database schema changes, use Recreate to kill v1 before v2 runs.
  • The main Multi-Attach deadlock solution — The Recreate strategy cleanly releases exclusively-accessed Persistent Volumes (PVs) first before re-attaching them to new Pods.
  • Do image pre-pulls to minimize downtime — Use a DaemonSet prefetcher to download images to worker nodes before the transition starts, shortening the service interruption period.
  • Configure automatic maintenance pages — Leverage the Ingress Controller’s default-backend feature to divert traffic to a user-friendly static maintenance page during downtime.
  • Optimize grace periods — Measure our application’s cleanup time and precisely adjust the terminationGracePeriodSeconds parameter so downtime isn’t extended.
  • Use for singleton workloads — The Recreate strategy guarantees two application instances never run simultaneously in the cluster at the same time.

← Previous: Canary Deployment   Next: Database Migration Strategy →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact