Deployment #

In large-scale system operations, one of the biggest challenges is how to update application code versions without disrupting users. Imagine if every time the developer team released a new feature, our web app or API had to suffer sudden downtime for several minutes. That would certainly damage business reputation. In Kubernetes, to automate the application lifecycle without downtime, we rely on a main abstraction object called Deployment.

A Deployment is a higher-level controller designed specifically for managing stateless applications. It acts as a declarator orchestrating the creation and deletion of ReplicaSet objects behind the scenes. Understanding its internal architecture, release calculation parameters, and rollback strategies is the foundational pillar for building stable, secure CI/CD pipelines in production.

Why Choose the Deployment Abstraction? #

Before the Deployment object existed, Kubernetes administrators had to manage release cycles manually using ReplicationController or ReplicaSet objects.

This was very error-prone because:

• Manual Updates: We had to manually scale down the old controller and scale up a new one through a series of CLI commands prone to network disconnects.
• Bypassed Rollbacks: If the new version crashed in a loop, there was no automatic way to roll back to the previous version except rewriting the original YAML manifest.

Deployment solves all those problems by providing a centralized lifecycle declaration:

1. Self-Healing: If a Pod dies from a runtime error or a node hangs, the Deployment immediately detects the failure and coordinates with the ReplicaSet to create a new Pod.
2. Zero-Downtime Updates: Performs gradual rolling updates guaranteeing there’s always an active application replica serving traffic during the update process.
3. Declarative Rollback: Stores configuration change history (revision history) so we can return the cluster to a previous healthy state within seconds.

Deployment Manifest Structure in Depth #

To leverage all the automation features above, we must write the Deployment manifest with the right spec parameters. Here’s a production-grade Deployment manifest example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api-deploy
  namespace: production
  labels:
    app.kubernetes.io/name: payment-api
spec:
  replicas: 4                     # Target number of Pod replicas guaranteed active
  revisionHistoryLimit: 10        # Number of old ReplicaSets kept in etcd
  progressDeadlineSeconds: 600    # Update timeout before being declared failed (10 minutes)
  selector:
    matchLabels:
      app: payment-api            # The connecting key for managing Pods
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%               # Maximum extra Pods during the update
      maxUnavailable: 0           # Maximum dead Pods during the update
  template:
    metadata:
      labels:
        app: payment-api          # MUST exactly match the selector above
    spec:
      containers:
      - name: api-container
        image: payment-api:v2.1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 5

Key Spec Parameters: #

• revisionHistoryLimit: Limits the number of empty ReplicaSet objects (0 capacity) stored in etcd. It’s recommended to set the default value 10 so we have enough rollback history without bloating the etcd database size.
• progressDeadlineSeconds: Sets the maximum time limit for the Kubelet to complete the new Pod startup process. If the new Pod fails the readiness probe within that time, the Deployment Controller changes its condition status to False with the reason ProgressDeadlineExceeded, triggering a failure notification to the monitoring system.

The Zero-Downtime Release Strategy: Rolling Update #

RollingUpdate is Kubernetes’ default release strategy, gradually replacing old Pod versions with new Pod versions. The release pipeline is controlled by two critical parameters: maxSurge and maxUnavailable.

Here’s a visualization of the gradual v1-to-v2 Pod transition with a target of 4 replicas, maxSurge: 25% (1 extra Pod), and maxUnavailable: 0:

flowchart TD
    subgraph Step0["Initial State (v1 Active)"]
        P1["v1 Pod"] & P2["v1 Pod"] & P3["v1 Pod"] & P4["v1 Pod"]
    end
    
    subgraph Step1["Step 1: Surge a v2 Pod"]
        P1 & P2 & P3 & P4
        S1["v2 Pod (Surge)"]
    end
    
    subgraph Step2["Step 2: v2 Ready, Delete a v1"]
        P1 & P2 & P3
        S1
    end
    
    subgraph Step3["Step 3: Surge a Second v2 Pod"]
        P1 & P2 & P3
        S1 & S2["v2 Pod"]
    end

    subgraph Step4["Final State (Full v2)"]
        S1 & S2 & S3["v2 Pod"] & S4["v2 Pod"]
    end

    Step0 -->|Create 1 New Pod| Step1
    Step1 -->|v2 Passes Readiness, Delete 1 v1| Step2
    Step2 -->|Create the Next v2| Step3
    Step3 -->|Repeat until v1 Is Gone| Step4

Choosing the Right Release Parameter Combination #

We can configure maxSurge and maxUnavailable values using absolute integers (e.g. 1) or percentages (e.g. 25%). Choosing values involves these trade-offs:

The Recreate Release Strategy: When Is Downtime Needed? #

The Recreate strategy works drastically: it immediately kills all old-version Pods at once, waits until they’re fully dead, then creates the new-version Pod group simultaneously. This action automatically triggers downtime (inaccessible service) during the container transition window.

Even though it causes downtime, the Recreate strategy must be chosen in these scenarios:

1. Non-Concurrent Applications: Our application isn’t designed to run side by side across versions (dual-version incompatibility). For example, new code writes database schema that crashes when accessed by old code.
2. Network IP/Port Constraints: The application uses the same host port (hostPort), so new-version containers can’t start before old containers free up that network port.

Release Lifecycle in the Field: CLI Commands #

In real production environments, changes are recommended to follow a GitOps approach (changing YAML files in Git). However, for quick experiments in development/staging environments, we can rely on the following CLI commands:

# 1. Directly update the application image
kubectl set image deployment/payment-api-deploy api-container=payment-api:v2.2.0

# 2. Monitor the update process in real-time
kubectl rollout status deployment/payment-api-deploy

# 3. View the revision change history
kubectl rollout history deployment/payment-api-deploy

# 4. Add an audit note so the history is easy to read
kubectl annotate deployment/payment-api-deploy \
  kubernetes.io/change-cause="Security Feature Release: Update image to v2.2.0 to fix CVE-2026-xxx"

The Pause & Resume Technique (Simple Manual Canary) #

If we want to do a simple Canary release without a service mesh, we can hold the update process mid-way to test on a small fraction of user traffic:

# Change the image to trigger an update
kubectl set image deployment/payment-api-deploy api-container=payment-api:v2.2.0

# Immediately pause the update process (after 1 or 2 new Pods are up)
kubectl rollout pause deployment/payment-api-deploy

# Run tests. Some traffic goes to new Pods, some to old Pods.
# If a bug is found, cancel and roll back:
kubectl rollout undo deployment/payment-api-deploy

# If all is safe, resume the release process until done:
kubectl rollout resume deployment/payment-api-deploy

Disaster Recovery Using Rollback #

If our application suddenly suffers performance degradation or a memory leak after a version update completes, we must take immediate rescue action to restore the service to its previous stable state.

# Roll back directly to the LAST REVISION
kubectl rollout undo deployment/payment-api-deploy

# Roll back to a specific past REVISION (e.g. revision 2)
kubectl rollout undo deployment/payment-api-deploy --to-revision=2

This rollback cycle runs very fast because the Deployment Controller only needs to reverse the ReplicaSet synchronization direction: raising the healthy old ReplicaSet’s capacity back to the target number, and dropping the problematic new ReplicaSet’s capacity to 0.

Diagnosing Stuck or Failed Releases #

One classic Kubernetes problem is a rolling update stuck mid-way. This is usually marked by the kubectl rollout status command hanging indefinitely.

Systematic Diagnosis Steps: #

1. Find the Problematic Pod:

   kubectl get pods -l app=payment-api
   

   Note whether any new Pods have ImagePullBackOff status (wrong image name/tag) or CrashLoopBackOff (app crashing during boot due to wrong config).
2. Analyze Kubelet Events:

   kubectl describe pod <failing-new-pod-name>
   

   Reading the event logs at the bottom of the describe output reveals whether the node ran out of memory or failed to pull the image from a private registry.
3. Check Application Logs:

   # If the container keeps crashing, read the logs from the previous execution
   kubectl logs <pod-name> --previous
   

Anti-Patterns in Deployment Management #

Deployment configuration mistakes that often cause operational accidents in production clusters:

Anti-Pattern 1: Skipping the Readiness Probe in Rolling Updates #

Deploying applications without a container readiness testing mechanism.

ANTI-PATTERN: Not Writing readinessProbe in the Deployment Manifest
// WHAT WE DO:
- Run a rolling update of the app image from `:v1` to `:v2`.
- In the container template, we don't write the `readinessProbe` configuration.
- The `:v2` image turns out to be broken and crashes (exit code 1) within 5 seconds of booting.

// THE CONSEQUENCES IN PRODUCTION:
- Total Downtime: Without a readiness probe, Kubernetes assumes new Pods are instantly "healthy"
  as soon as the container starts (even if the app hasn't finished loading).
- The Deployment Controller keeps blindly deleting old `:v1` Pods.
- Within minutes, all healthy `:v1` Pods are deleted and replaced by crashing `:v2` Pods.
- The service is completely paralyzed.

✓ THE RIGHT SOLUTION:
- Always provide a dedicated health check endpoint (e.g. `/healthz/ready`) in your application code.
- Configure `readinessProbe` accurately in the Deployment manifest.
- Kubernetes is guaranteed to never kill old Pods before new Pods truly pass the readiness probe.

Anti-Pattern 2: Using Large-Scale Imperative Scripts for Updates #

Relying on chains of manual CLI commands in the CI/CD pipeline.

ANTI-PATTERN: Chaining kubectl scale and kubectl set image
// WHAT WE DO:
- Write a bash script in Jenkins/GitLab CI:
  `kubectl set image deployment/my-deploy app=my-image:tag && kubectl scale deployment/my-deploy --replicas=10`

// THE CONSEQUENCES IN PRODUCTION:
- Losing the Real State: Imperative commands trigger conflicts if run simultaneously from two different pipelines.
- If the git repository manifest isn't updated, when GitOps (like ArgoCD) performs automatic disaster recovery,
  all manual configuration in the production cluster gets wiped and overwritten back to the stale config in the Git repo.

✓ THE RIGHT SOLUTION:
- Hold firmly to the GitOps principle.
- Update the image tag or replica count directly in the YAML manifest file in the Git repository.
- Apply changes to the cluster using a single declarative `kubectl apply -f deployment.yaml`.

Summary #

• The Main Stateless Abstraction — Deployment is the primary controller for stateless applications, managing automatic ReplicaSet creation, scaling, and deletion.
• Zero-Downtime via RollingUpdate — Use the RollingUpdate strategy to update application versions gradually without taking the service down for users.
• The maxSurge and maxUnavailable Rules — Tune release parameters carefully: maxUnavailable: 0 guarantees no downtime, while maxSurge controls the extra memory capacity needs during the update.
• Recreate for Incompatibility — Use the Recreate strategy only if old and new application versions must never be active simultaneously against the same database.
• Instant History-Based Rollback — Deployments keep old ReplicaSet objects to enable instant rollback via kubectl rollout undo.
• Audit Capability via Change-Cause — Use the kubernetes.io/change-cause annotation on Deployments to record release reason history for DevOps team audits.
• Readiness Probes Are Mandatory — Never run a rolling update without defining readinessProbe to prevent broken containers from replacing healthy ones.
• Apply the Declarative Principle — Avoid imperative modifications (kubectl scale/set image) in production; update the Git YAML manifest files to maintain a single source of truth.

← Previous: ReplicaSet   Next: StatefulSet →