Deployment Strategy Overview #

Every time we launch a new version of an application to production, we’re introducing risk into our system. New code may contain hidden bugs, database query performance can drastically degrade under real traffic load, or API behavior changes can accidentally break integrations with other services. A deployment strategy is the structured methodology we use to manage and mitigate those risks. The strategy we choose determines how big the failure blast radius is if a new release has problems, how fast we can return the system to a stable condition (rollback), and how much infrastructure cost we must bear during the release process.

Kubernetes is designed modularly to support various release strategies, from the simplest to complex progressive delivery systems. However, no single strategy is universal and best for all application types. Choosing the right deployment strategy requires understanding our application workload characteristics, business downtime tolerance, database readiness to support multi-version concurrency, and the cluster’s available resource capacity.

The Trade-Off Spectrum: Risk vs Complexity #

When designing a deployment strategy, we’re always faced with a spectrum of conflicting trade-offs. We can’t get instant rollback and zero downtime without paying the price of increased infrastructure complexity and resource waste.

In general, the relationship between the main deployment strategies can be illustrated as follows:

flowchart TD
    subgraph Spec["DEPLOYMENT STRATEGY SPECTRUM"]
        direction LR
        Canary["Canary<br>- Gradual release to a small subset of users<br>- Downtime & Risk: Lower<br>- Complexity & Resource: Higher"]
        BlueGreen["Blue/Green<br>- Two parallel environments, instant cutover<br>- Downtime & Risk: Low<br>- Complexity & Resource: High"]
        RollingUpdate["Rolling Update<br>- One-by-one Pod replacement, k8s default<br>- Downtime & Risk: Medium<br>- Complexity & Resource: Low"]
        Recreate["Recreate<br>- Kill all v1, run all v2<br>- Downtime & Risk: Higher<br>- Complexity & Resource: Lower"]

        Canary --> BlueGreen --> RollingUpdate --> Recreate
    end

1. Exposure Risk: Shows how many users are directly impacted if new code has critical bugs when first released.
2. Resource Footprint: How much extra CPU and Memory capacity is needed in the cluster during the release transition process.
3. Operational Complexity: The difficulty level of configuring the deployment pipeline (CI/CD), monitoring metrics, and automating the rollback process.
4. Rollback Speed: The time needed to fully return the system to a stable version (rollback window) if anomalies are detected.

Dissecting the Four Main Deployment Strategies #

Kubernetes provides internal mechanisms for managing release strategies through the .spec.strategy property on Deployment objects. Let’s dissect the architectural characteristics of each approach:

1. Recreate Strategy #

The Recreate strategy is the most primitive yet easiest to understand approach. Kubernetes terminates all old version (v1) Pod replicas simultaneously until the count reaches zero, waits until all volumes are released, and only then creates and runs all new version (v2) Pod replicas.

[ TIME ] ───►
T1: v1 Pods (Active) [ Pod 1 (v1) ] [ Pod 2 (v1) ] [ Pod 3 (v1) ]
T2: v1 Termination    [ Off (v1) ]   [ Off (v1) ]   [ Off (v1) ]     ◄─── DOWNTIME WINDOW
T3: v2 Startup        [ Init (v2) ]  [ Init (v2) ]  [ Init (v2) ]    ◄─── DOWNTIME WINDOW
T4: v2 Pods (Active)  [ Pod 1 (v2) ] [ Pod 2 (v2) ] [ Pod 3 (v2) ]

• Downtime: Yes. There’s a clear time gap (downtime window) between when v1 Pods die and when v2 Pods are ready to receive traffic.
• Extra Resources: 0%. No additional node capacity is needed because new Pods occupy the resource slots left by old Pods.
• Rollback: Slow. We must repeat the v2 termination process and wait for v1 startup from scratch.
• Best Use Cases:
  • Legacy monolith applications not designed to run more than one instance simultaneously (don’t support concurrent database locks).
  • Non-backward-compatible database schema changes where v1 would immediately crash when reading the new table structure.
  • Development/sandbox environments where a few minutes of downtime isn’t a business problem.

2. Rolling Update Strategy #

Rolling Update is the built-in (default) strategy in Kubernetes. Kubernetes gradually replaces old Pods with new Pods one at a time or in small groups. This process is governed by two main parameters: maxSurge (how many extra Pods may be created above the desired replica count) and maxUnavailable (how many Pods may be unavailable during the transition).

T1: Start            [ Pod 1 (v1) ] [ Pod 2 (v1) ] [ Pod 3 (v1) ]
T2: v2 Surge         [ Pod 1 (v1) ] [ Pod 2 (v1) ] [ Pod 3 (v1) ] + [ Pod 4 (v2) ]
T3: Pod 1 Termination [ Term (v1) ]  [ Pod 2 (v1) ] [ Pod 3 (v1) ] + [ Pod 4 (v2) ]
T4: Replacement      [ Pod 5 (v2) ] [ Pod 2 (v1) ] [ Pod 3 (v1) ] + [ Pod 4 (v2) ]
...
T_End: Done          [ Pod 5 (v2) ] [ Pod 6 (v2) ] [ Pod 7 (v2) ]

• Downtime: No. Service availability stays maintained during the update process because there are always Pods ready to serve traffic.
• Extra Resources: Minimal (depending on the maxSurge configuration, usually ranging from 25% to 50%).
• Rollback: Medium (takes a few minutes for Kubernetes to do a reverse rolling update back to the previous version).
• Key Characteristic: During the transition, versions v1 and v2 run simultaneously (concurrent runtime) in the cluster. Our application must be designed to tolerate this condition (backward-compatible).

3. Blue/Green Deployment (Red/Black) #

In the Blue/Green strategy, we run two separate physically identical environments in the cluster. The “Blue” environment is the currently active production version (v1) serving all user traffic. The “Green” environment is the new version (v2) we deploy fully alongside the Blue environment.

We can do thorough smoke testing on the Green environment without disturbing real user traffic. After the Green environment is confirmed healthy and passes tests, we instantly move (cutover) all traffic to the Green environment by changing the reference in the Service selector or routing rules in the Ingress/Gateway API.

flowchart TD
    subgraph Before["Before Cutover"]
        Traffic1["Traffic"] --> Ingress1["Service / Ingress"]
        Ingress1 --> Blue1["BLUE environment (Pod v1)"]
        Ingress1 -.-> Green1["GREEN environment (Pod v2)"]
    end
    subgraph After["After Cutover"]
        Traffic2["Traffic"] --> Ingress2["Service / Ingress"]
        Ingress2 -.-> Blue2["BLUE environment (Standby v1)"]
        Ingress2 --> Green2["GREEN environment (Pod v2)"]
    end

• Downtime: No. Traffic movement happens instantly (sub-second) at the proxy/DNS routing level.
• Extra Resources: 100%. We need double resource capacity during the release process because both environments must run fully in parallel.
• Rollback: Instant. If the Green version fails, we just change the routing reference back to the Blue environment, which is still on standby.

4. Canary Deployment #

Canary Deployment takes its analogy from the canary birds coal miners used to detect toxic gas. In this strategy, we deploy the new version (v2) to a small subset of container Pods, then direct a small portion of real user traffic (e.g. 5% or 10%) to those Pods.

We closely monitor the Canary version’s performance using observability metrics (like HTTP error percentage, query latency, and CPU usage). If the metrics show good results, we gradually increase the traffic portion (25% -> 50% -> 100%) until the entire cluster runs version v2.

flowchart LR
    Traffic["Incoming Traffic (100%)"]
    Traffic -->|"90% Traffic"| Stable["v1 Pods (Stable)"]
    Traffic -->|"10% Traffic"| Canary["v2 Pods (Canary - Closely Monitored)"]

• Downtime: No.
• Extra Resources: Very Little (only needs capacity for the initial Canary Pods, growing as the release scales up).
• Rollback: Fast and Controlled. If the Canary Pods show errors, we just delete the Canary Pods or stop the traffic route toward them. Only a small portion of users experienced the bug’s impact.

The Decision Framework (Decision Tree) #

To help teams choose the most suitable deployment strategy for each application workload, we can refer to the following decision flow diagram:

flowchart TD
    Start["Identify Application & Business Characteristics"] --> Q1{"Does the app support<br>two versions running simultaneously<br>'(Concurrent Database Lock/State)'?"}
    
    Q1 -- "No (Monolith / Rigid Schema)" --> Q2{"Does the business tolerate<br>brief downtime?"}
    
    Q2 -- "Yes" --> RecreateObj["CHOOSE: RECREATE STRATEGY"]
    Q2 -- "No" --> BlueGreenObj["CHOOSE: BLUE/GREEN DEPLOYMENT<br/>'(Instant Cutover)'"]
    
    Q1 -- "Yes (Stateless / Backward Compatible)" --> Q3{"How big is the release risk<br>and the SLA demands?"}
    
    Q3 -- "Low Risk / Routine Releases" --> RollingObj["CHOOSE: ROLLING UPDATE<br/>'(Kubernetes Default)'"]
    Q3 -- "High Risk / Major Features" --> Q4{"Does the infrastructure have<br>extra resource capacity (CPU/RAM)?"}
    
    Q4 -- "Limited" --> CanaryReplica["CHOOSE: CANARY (Pod Replica Weighting)"]
    Q4 -- "Sufficient" --> CanaryIngress["CHOOSE: CANARY (Ingress Traffic Shifting)<br/>or BLUE/GREEN"]
    
    style RecreateObj stroke:#f57c00,stroke-width:2px
    style BlueGreenObj stroke:#0288d1,stroke-width:2px
    style RollingObj stroke:#388e3c,stroke-width:2px
    style CanaryReplica stroke:#f57c00,stroke-width:2px
    style CanaryIngress stroke:#388e3c,stroke-width:2px

Strategy Comparison Matrix #

The following comparative table summarizes all technical and operational parameters for each strategy:

Anti-Patterns vs Best Solutions #

Let’s study some fatal mistakes that often happen due to wrong deployment strategy selection or implementation, along with the best solutions.

Anti-Pattern 1: Rolling Update on Stateful Applications with Exclusive Locked Volumes (ReadWriteOnce) #

Doing a rolling update on an application using a Persistent Volume (PV) with ReadWriteOnce (RWO) access mode on a cloud provider. When Kubernetes tries running v2 Pods before killing v1 Pods (due to maxSurge settings), the v2 Pods fail to start because the disk volume is still locked by v1 Pods.

v2 Pods stuck in status: ContainerCreating
Error Log: "Multi-Attach error for volume ... Volume is already used by Pod v1"

Best Solution #

Use the Recreate strategy to guarantee v1 Pods are fully released from the host node and the volume is detached before v2 Pods try to attach that volume.

# ✓ SOLUTION: Recreate strategy configuration for volume-locking stateful apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database-app
  namespace: database
spec:
  replicas: 1
  # Using Recreate to ensure a clean disk volume release
  strategy:
    type: Recreate
  template:
    spec:
      containers:
      - name: db
        image: postgres:15-alpine
        volumeMounts:
        - name: db-data
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: db-data
        persistentVolumeClaim:
          claimName: postgres-pvc # PVC with ReadWriteOnce mode

Anti-Pattern 2: Doing a Rolling Update Without Defining Readiness Probes #

Launching an application update using the Rolling Update strategy but leaving the container readinessProbe property empty. Kubernetes assumes the container is immediately ready to receive traffic as soon as the main process runs (Running status). As a result, Kubernetes kills old Pods (v1) while new Pods (v2) have just started initializing and aren’t truly ready to serve queries. This triggers massive production downtime even though we’re using the Rolling Update strategy.

# ✗ ANTI-PATTERN: Rolling update without a readiness probe
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
  template:
    spec:
      containers:
      - name: app
        image: company/payment-api:v2.0.0 # ← If initialization takes 20 seconds, users get 502/503 errors

Best Solution #

Always define accurate readinessProbe and livenessProbe so Kubernetes delays killing old Pods until new Pods truly return a success status (200 OK) on the application health endpoint.

# ✓ SOLUTION: Apply health probes to guarantee zero-downtime rolling updates
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%          # Maximum 1 extra Pod created above replicas (total 5)
      maxUnavailable: 25%    # Maximum 1 Pod may be inactive during transition
  template:
    spec:
      containers:
      - name: app
        image: company/payment-api:v2.0.0
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 15 # Give the app time to load libraries
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          periodSeconds: 10

Anti-Pattern 3: Running Blue/Green Releases Without Smoke Testing #

Using the Blue/Green strategy and switching the Service selector route directly right after the Green Pods reach Running status, without doing specific verification or functional testing on the Green environment.

Best Solution #

Prepare a special testing Service (test/smoke service) that exclusively points to the Green environment’s container labels. The QA team or our CI/CD pipeline must run automated testing scripts (smoke tests) against that testing Service first. After the tests pass, only then do we update the main production Service to point to the Green environment.

# ✓ SOLUTION: An Isolated Testing Service (Staging/Smoke Service)
apiVersion: v1
kind: Service
metadata:
  name: payment-api-test-service
  namespace: payment
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: payment-api
    # Points exclusively to the Green environment being tested
    version: v2.0.0-green 

Implementation Plans for This Section #

In the rest of this chapter, we’ll learn the tactical implementation of each concept above:

• Rolling Update: Mathematically understanding surge/unavailable limit calculations, probe optimization, and rollout CLI control.
• Blue/Green Deployment: A guide to building dual environments with Ingress weighting, Service updates, and database session management.
• Canary Deployment: Implementing advanced traffic shifting using Service Mesh and release automation with Argo Rollouts.
• Recreate: Avoiding attach-detach disk deadlocks on clusters and minimizing release downtime windows.
• Database Migration Strategy: Orchestrating database schemas without downtime using the Expand-Contract pattern and Kubernetes Jobs.
• Rollback Strategy: Handling automatic post-release-failure recovery based on anomaly metrics.
• GitOps: Declaring cluster state using a Git repository as the production release control center.
• Deployment Anti-Patterns: Evaluating real industry deployment failure cases to prevent downtime in our clusters.

Summary #

• There’s no universal best strategy — Deployment strategy selection is the result of weighing trade-offs between business downtime tolerance, failure risk, and infrastructure cost.
• Use Rolling Update for routine updates — This Kubernetes built-in strategy guarantees zero-downtime and resource efficiency, but demands our application code be compatible when v1 and v2 run simultaneously.
• Choose Recreate for rigid DB schemas and Stateful apps — If the application doesn’t support database concurrency or exclusively locks volumes, use Recreate to avoid volume attach failures.
• Apply Blue/Green for instant rollback — By maintaining two identical environments in parallel, we gain the ability to flip traffic back to the stable version in sub-seconds if the new release breaks.
• Maximize blast radius minimization with Canary — Gradually channel real traffic to the new version to limit release failure impact to only a small initial user subset.
• Must install Probes — All zero-downtime strategies (Rolling Update, Blue/Green, Canary) won’t work if we don’t configure accurate readinessProbes on application containers.

← Previous: Configuration Anti-Patterns   Next: Rolling Update →