Deployment Anti-Patterns #

Launching code changes to production is the most crucial yet failure-prone activity in software engineering. Kubernetes provides various robust abstraction objects to help us design downtime-free release strategies. However, this infrastructure flexibility is often misused due to a lack of deep understanding of how Kubernetes interacts with worker node file systems, how network traffic routing is dynamically updated, and how orchestrated database queries run during container bootstrap.

Often, deployment misconfigurations look normal when tested in Development or Staging environments because of low traffic volume and the absence of node resource restrictions. However, once that broken release lands in Production under high real user traffic load, operational disaster immediately strikes. This article deeply reviews the eight deployment anti-patterns most often triggering incidents in production Kubernetes clusters, analyzes their failure impacts, and presents the best declarative manifest solutions to fix them.

Deployment Impact Risk Matrix #

Before discussing each mistake in depth, let’s review the risk assessment matrix flow diagram below to help identify which parts of our deployment architecture manifests are still vulnerable to failure:

flowchart TD
    Start["Audit Production Deployment Manifests"] --> Q1{"Are there Services without<br>a 'readinessProbe' definition?"}
    
    Q1 -- "Yes" --> RiskReady["HIGH RISK:<br>HTTP 502/503 errors during Rolling Updates"]
    Q1 -- "No" --> Q2{"Do container images use<br>the dynamic 'latest' tag?"}
    
    Q2 -- "Yes" --> RiskTag["HIGH RISK:<br>Rollback impossible & non-deterministic releases"]
    Q2 -- "No" --> Q3{"Are DB migrations executed<br>via initContainers in the Deployment?"}
    
    Q3 -- "Yes" --> RiskMigrate["MEDIUM-HIGH RISK:<br>Race conditions & database table locks"]
    Q3 -- "No" --> Q4{"Do containers run without<br>resources limits/requests configuration?"}
    
    Q4 -- "Yes" --> RiskOOM["MEDIUM RISK:<br>Pod OOMKilled & cluster instability"]
    Q4 -- "No" --> SafeDeployment["Stable Deployment Following Best Practices"]
    
    style RiskReady stroke:#d32f2f,stroke-width:2px
    style RiskTag stroke:#d32f2f,stroke-width:2px
    style RiskMigrate stroke:#f57c00,stroke-width:2px
    style RiskOOM stroke:#f57c00,stroke-width:2px
    style SafeDeployment stroke:#388e3c,stroke-width:2px

Anti-Pattern 1: Using the latest Image Tag in Production #

One of the most fundamental yet operationally fatal mistakes is referencing application container images with the dynamic latest tag in production Deployment manifests.

# ✗ ANTI-PATTERN: Using the latest image tag on production containers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
      - name: api
        image: company/order-api:latest # ← CRITICAL: Dynamic tags aren't deterministic
        imagePullPolicy: Always # ← Forces image pulls on every node startup (burdens the network)

Why Is This Dangerous? #

1. Rollback Failures: If the new version pushed to the registry turns out broken, the kubectl rollout undo command fails to restore the cluster’s stable state. Kubernetes only tries re-pulling the same latest-labeled image from the registry, so we stay stuck on the same broken version.
2. Non-Deterministic Releases: We lose a clear audit trail. We can’t answer with certainty: “Which specific code version is currently running in production?”
3. Worker Node Inconsistency (Split-Brain): If our cluster has many worker nodes, one worker node might use a latest image version cached 2 hours ago, while another worker node pulls the latest version pushed 5 minutes ago. Pod replicas running in one cluster exhibit different code behaviors.

Best Solution #

Use unique, static, immutable image tags like Git commit hashes (commit SHA) or SemVer (Semantic Versioning) numbers. Set the imagePullPolicy property to IfNotPresent to save cluster network bandwidth.

# ✓ SOLUTION: Using an immutable image tag based on a Git commit
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
      - name: api
        image: company/order-api:sha-8f8d2f1 # Pin a clear Git commit hash
        imagePullPolicy: IfNotPresent # Only download if the image isn't already on the node

Anti-Pattern 2: Omitting the readinessProbe Definition on Containers #

Many developer teams only define a livenessProbe (to detect crash loops) but leave the readinessProbe property empty on their Deployment manifests.

# ✗ ANTI-PATTERN: Deployment without a readinessProbe definition
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: company/billing-service:v2.0.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
        # No readinessProbe validating database connection socket readiness

Why Is This Dangerous? #

As soon as the container’s main process runs, Kubernetes by default considers the new Pod healthy (Ready) and adds it to the Service traffic routing list. In reality, the application inside the container may need 10-15 seconds to initialize libraries, load database connections, or warm up caches.

Real user traffic diverted to that new Pod gets dropped immediately, producing periodic HTTP 502 Bad Gateway or 503 Service Unavailable error spikes during the Rolling Update cycle.

Best Solution #

Always provide a special health endpoint (e.g. /healthz/ready) dynamically verifying whether the application is ready to receive queries (like checking active database and Redis connections).

# ✓ SOLUTION: Accurate readinessProbe configuration for zero-downtime
spec:
  containers:
  - name: app
    image: company/billing-service:v2.0.0
    readinessProbe:
      httpGet:
        path: /healthz/ready # Special endpoint validating backend database/cache readiness
        port: 8080
      initialDelaySeconds: 10 # Initial application initialization wait time
      periodSeconds: 5
      failureThreshold: 3

Anti-Pattern 3: Deployments Without Resource Limits (No Requests/Limits) #

The act of launching Pods into the cluster without configuring the resources.requests and resources.limits properties.

# ✗ ANTI-PATTERN: Containers running without CPU/RAM resource limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: processing-worker
spec:
  template:
    spec:
      containers:
      - name: worker
        image: company/processing-worker:v1.0.0
        # The resources block is omitted

Why Is This Dangerous? #

1. Scheduling Blindness: The Kubernetes scheduler doesn’t know how much real CPU and RAM capacity the Pod needs. Pods can be placed on worker nodes that are actually out of memory.
2. Noisy Neighbor Effect: If a memory leak occurs in one Pod, that Pod can suck up the worker node’s entire physical RAM capacity, force-killing (OOMKilled) all other innocent Pods on that node.
3. Stuck Deployments: During Rolling Updates, creating additional new Pods (maxSurge) can fail if the worker node has no formally recorded leftover resources, causing the release to stall without clarity.

Best Solution #

Benchmark our application’s CPU and RAM usage under peak load, then precisely set requests (for scheduling) and limits (for protection) values.

# ✓ SOLUTION: Strictly define CPU/RAM resource limits
spec:
  containers:
  - name: worker
    image: company/processing-worker:v1.0.0
    resources:
      requests:
        cpu: "250m"
        memory: "256Mi"
      limits:
        cpu: "1000m"
        memory: "512Mi" # Prevents Pods from eating memory beyond safe host limits

Anti-Pattern 4: Executing Database Migrations from initContainers #

Embedding database schema migration queries (like django-admin migrate or flyway migrate commands) into the initContainers block of application Deployment manifests.

# ✗ ANTI-PATTERN: Running database migration scripts in initContainers
spec:
  replicas: 5
  template:
    spec:
      initContainers:
      - name: db-migrate
        image: company/order-api:v2.0.0
        command: ["python", "manage.py", "migrate"] # ← DANGEROUS: Executed in parallel at startup
      containers:
      - name: app
        image: company/order-api:v2.0.0

Why Is This Dangerous? #

If our Deployment has 5 Pod replicas, when the Rolling Update process starts, all five initContainers turn on in parallel and try executing the same database migration script to one database server simultaneously.

This triggers race conditions, migration metadata table locking (table lock deadlocks), duplicate query failures, and mass crash loops paralyzing new Pod startup processes.

Best Solution #

Separate the database migration lifecycle. Run database migration scripts using a separate Kubernetes Job object guaranteeing single query execution (run-to-completion) before the application Deployment is updated.

# ✓ SOLUTION: Run database migrations via a standalone Kubernetes Job
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-v2-0-0
  namespace: production
spec:
  completions: 1
  parallelism: 1 # Guarantees only 1 migration Pod runs
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: migrate
        image: company/order-api:v2.0.0
        command: ["python", "manage.py", "migrate"]

Anti-Pattern 5: Omitting preStop Hooks for Graceful Shutdown #

Letting high-traffic HTTP web containers die instantly upon receiving the SIGTERM signal from the Kubelet, without any network propagation delay time.

# ✗ ANTI-PATTERN: The app dies instantly without network routing propagation handling
spec:
  terminationGracePeriodSeconds: 5 # The grace period is too short
  template:
    spec:
      containers:
      - name: web-app
        image: company/web-app:v1.0.0
        # preStop hook not configured

Why Is This Dangerous? #

When the Kubelet sends the SIGTERM signal to the container to start Pod shutdown, Kubernetes in parallel starts updating routing tables on the Ingress Controller and kube-proxy. However, the routing synchronization process across worker nodes takes propagation time (usually 2-5 seconds).

If the container immediately stops serving connections upon receiving SIGTERM, the remaining user request traffic sent during that propagation gap hits empty sockets, producing HTTP 502 Bad Gateway errors in user browsers.

sequenceDiagram
    participant User as HTTP User
    participant Ingress as Ingress Controller
    participant Pod as v1 Pod (Terminating)
    participant Kubelet as Worker Node Kubelet
    
    Kubelet->>Pod: Send the SIGTERM signal
    Pod->>Pod: Close sockets & kill the process instantly!
    Note over Ingress: Routing table sync process still running (2-5 second delay)
    User->>Ingress: Send a new request
    Ingress->>Pod: Forward the request to the v1 Pod IP
    Pod-->>Ingress: Connection Refused / Drop
    Ingress-->>User: HTTP 502 Bad Gateway!

Best Solution #

Add a preStop lifecycle hook on the container forcing it to delay (sleep) for a few seconds before allowing the SIGTERM signal to be sent to the application.

# ✓ SOLUTION: Adding a preStop hook to guarantee zero-downtime rolling updates
spec:
  terminationGracePeriodSeconds: 30 # Give enough time to clean up transactions
  template:
    spec:
      containers:
      - name: web-app
        image: company/web-app:v1.0.0
        lifecycle:
          preStop:
            exec:
              # Gives the network routing tables time to fully update
              command: ["/bin/sh", "-c", "sleep 10"]

Anti-Pattern 6: Direct Cluster Modifications (Bypassing GitOps / Hotpatching) #

Cluster operators or developers run kubectl edit, kubectl patch, or kubectl set image commands directly on the production cluster to apply quick fixes (hotfixes) in the middle of the night.

# ✗ ANTI-PATTERN: Manually editing the production image directly in the cluster
kubectl set image deployment/payment-gateway gateway=company/gateway:v1.2.1-hotfix -n payment

Why Is This Dangerous? #

1. Configuration Drift: The cluster’s actual state has deviated from the Git repository state.
2. Instant Reversion (Overriding): If we install a GitOps operator (like ArgoCD) with the selfHeal: true feature, ArgoCD detects our manual change as unauthorized deviation. Within seconds, ArgoCD overwrites our manual modification and returns the cluster to the stable version in the Git repository, instantly killing our emergency fix.
3. Lost Audit Trails: Emergency changes aren’t recorded in the company repository’s Git commit history, complicating root cause analysis.

Best Solution #

All configuration and image version changes must go through the Git gate (Git-centric workflow). Commit to the GitOps repository, merge into the release branch, and let the GitOps operator execute it to the cluster safely.

Anti-Pattern 7: Setting revisionHistoryLimit to Zero (0) #

Trying to minimize the cluster’s etcd database resources by disabling old ReplicaSet history storage.

# ✗ ANTI-PATTERN: Disabling ReplicaSet revision history storage
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  revisionHistoryLimit: 0 # ← CRITICAL: All old ReplicaSets deleted instantly after release

Why Is This Dangerous? #

With a zero value, Kubernetes stores no previous version backup metadata. If a critical bug appears post-new-release, the kubectl rollout undo command can’t be used. We’re forced to wait for the CI/CD pipeline to rebuild the old image or manually rewrite the manifest from scratch, drastically extending downtime duration during incidents.

Best Solution #

Configure the revisionHistoryLimit value to at least 5 to guarantee sufficient recovery backups in the cluster.

# ✓ SOLUTION: Maintain a safe revision history limit
spec:
  revisionHistoryLimit: 10 # Keeps the last 10 backup revisions for instant rollback

Anti-Pattern 8: The maxSurge: 0 and maxUnavailable: 0 Deadlock Configuration #

A Rolling Update transition parameter writing error where both parameters are limited to zero.

# ✗ ANTI-PATTERN: An invalid transition configuration creating a deadlock
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0        # ← Not allowed to create extra new Pods
      maxUnavailable: 0  # ← Not allowed to kill old Pods

Why Is This Dangerous? #

Although modern Kubernetes API Server versions reject this manifest at the schema validation stage, if it enters the controller, this configuration triggers an operational deadlock. Kubernetes can’t create new Pods (because maxSurge is limited to zero) and can’t kill old Pods to free scheduling slots (because maxUnavailable is limited to zero). The Deployment update process stalls forever.

Best Solution #

Use a valid value combination. The most recommended production zero-downtime standard configuration is maxSurge: 25% (or 1) and maxUnavailable: 0.

# ✓ SOLUTION: A safe alternating transition parameter combination
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0

Best Production Deployment Manifest (Anti-Pattern Hardened) #

Here’s a Deployment manifest example hardened against all the anti-patterns above, ready for critical production workloads:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-service
  namespace: core
  annotations:
    # Clear git commit hash recording for audit trails
    kubernetes.io/change-cause: "Upgrade to version v2.2.0 - Git SHA a8c3f2d"
spec:
  replicas: 4
  minReadySeconds: 15
  progressDeadlineSeconds: 300
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: catalog-service
  template:
    metadata:
      labels:
        app: catalog-service
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: catalog-app
        image: company/catalog-app:v2.2.0 # Pins a clear static version tag (SemVer)
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "1000m"
            memory: "512Mi" # Protects against memory leaks
        lifecycle:
          preStop:
            exec:
              # sleep 15 seconds to secure in-flight requests during routing transitions
              command: ["/bin/sh", "-c", "sleep 15"]
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3

Production Deployment Hardening Audit Checklist #

Make sure our cluster Deployment manifests avoid all critical configuration errors using the following checklist:

IMAGE & PROBE MANAGEMENT:
  □ Container image tags are pinned using Git commit hashes or unique SemVer (not 'latest').
  □ The 'imagePullPolicy' parameter is set to 'IfNotPresent' to save cluster network bandwidth.
  □ The 'readinessProbe' property is accurately defined on a valid backend readiness endpoint (like database checks).

GRACEFUL SHUTDOWN & NETWORK TRANSITIONS:
  □ The 'lifecycle.preStop' property is configured with a minimum 5-15 second sleep delay on HTTP web containers.
  □ The 'terminationGracePeriodSeconds' parameter has a value larger than the preStop hook sleep duration.
  □ The 'maxSurge' and 'maxUnavailable' parameters aren't both set to zero.

RESOURCES & DATA ORCHESTRATION:
  □ 'resources.requests' and 'resources.limits' objects are precisely configured to guarantee a healthy QoS Class.
  □ Database migration queries are exclusively moved to a separate Kubernetes Job object (not running in initContainers).
  □ The 'revisionHistoryLimit' parameter is configured to at least 5 to guarantee emergency rollback availability.

Summary #

• Pin container image versions — Never use the latest tag in production; use Git commit SHAs or SemVer to guarantee release reproducibility and smooth rollbacks.
• Must install readinessProbes — Health probes guarantee Kubernetes never sends user traffic to containers that haven’t finished initial initialization.
• Define CPU/RAM limits & requests — Prevents resource shortages on worker nodes and protects other containers from memory leak effects (Noisy Neighbor).
• Use Kubernetes Jobs for migrations — Don’t run database migration scripts in initContainers to avoid metadata deadlocks and race conditions between Pods.
• Apply preStop sleep hooks — Give the network routing tables time to sync terminating Pod IP data before the main process is force-killed.
• Maintain cluster revision history — Configure revisionHistoryLimit to at least 5 to guarantee backup ReplicaSet snapshot availability for production emergencies.

← Previous: GitOps   Next: RBAC (Role-Based Access Control) →