Production Anti-Patterns #

Most service outage incidents in production Kubernetes environments are rarely caused by unexpected external system failures or previously unidentified external bugs. The majority of system failures actually come from accumulations of dangerous configuration patterns (anti-patterns) ignored during development processes under the excuse “we’ll fix it later when we have time”. When our cluster grows large and user traffic loads increase, these small holes get triggered simultaneously, resulting in hard-to-isolate cascade failures. As the closing piece of this entire Kubernetes guide series, this article summarizes the seven most critical production anti-patterns often found in mature advanced clusters, deeply dissects their real operational impacts, and presents tactical mitigation solutions ready to be applied to secure our production cluster stability.

Anti-Pattern 1: Missing PodDisruptionBudgets (PDBs) #

When platform teams or cluster administrators do planned maintenance activities (like draining nodes to update host OS versions), Kubernetes moves Pods to other Nodes. Without PodDisruptionBudgets (PDBs), Kubernetes doesn’t care how many of our application replica pods are actively serving users.

# File: templates/deployment.yaml
# ANTI-PATTERN: A critical production application Deployment without an accompanying PDB
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-gateway
  namespace: prod-apps
spec:
  replicas: 3
  # DON'T: The missing PDB in your Helm release manifests!

Understanding Voluntary Disruptions Behavior #

Kubernetes distinguishes disruptions into two types:

• Involuntary Disruptions (Unavoidable): Physical Node hardware failures, network disconnections, or disk damage. These can’t be prevented by PDBs.
• Voluntary Disruptions (Intentional/Scheduled): Node draining (kubectl drain) by admins, Node removal by the Cluster Autoscaler, or manual Pod evictions. PDBs are specifically designed to protect applications from these scenarios.

When administrators run Node drain commands:

$ kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data

The drain command automatically does a cordon (marks the node as not accepting new Pods) before doing evictions. Without PDBs, if the three payment-gateway pods above are on node-worker-1 and node-worker-2, Kubernetes destroys those pods simultaneously. As a result, our service capacity drops to 0% while replacement pod creation processes run on new nodes, triggering instant transaction downtime for users.

Constructive Solutions #

Always define PDB objects together with critical application Deployments in our release manifests.

# File: templates/pdb.yaml
# CORRECT: Setting minimum disruption budgets for high availability
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-gateway-pdb
  namespace: prod-apps
spec:
  # Option 1: Guarantee at least 2 pods are always healthy during node maintenance
  minAvailable: 2 
  
  # Option 2: Or limit the maximum pods allowed down simultaneously
  # maxUnavailable: 1 
  
  selector:
    matchLabels:
      app: payment-gateway

Anti-Pattern 2: Enabling HPA Without Requests Allocations #

HPAs are designed to dynamically scale pods based on CPU or memory usage percentages. However, a common anti-pattern is teams installing HPAs without defining resources.requests blocks at the Deployment container level.

# File: k8s/production-hpa-conflicted.yaml
# HPA configured with a 60% CPU target
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: customer-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-api
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

# DON'T: The target Deployment has no requests.cpu!
spec:
  template:
    spec:
      containers:
      - name: customer-api
        resources:
          # Limits are defined, but CPU requests are left empty!
          limits:
            cpu: "1000m"
            memory: "512Mi"

Operational Consequences and HPA Logic Failures #

HPAs work by calculating utilization percentages based on the formula:

$$\text{CPU Utilization Percentage} = \left( \frac{\text{Active CPU Usage}}{\text{CPU Requests}} \right) \times 100%$$

When requests.cpu isn’t defined, the divisor in the formula above is null. As a result, the HPA Controller inside the kube-controller-manager throws a FailedGetResourceMetric error because it can’t calculate the Pod’s utilization percentage.

If we check the HPA status using the command:

$ kubectl get hpa customer-api-hpa

Then the TARGETS column permanently displays the <unknown>/60% value. When massive traffic surges happen, the HPA suffers scale paralysis, doesn’t trigger new replica creation, and lets our single pod crash from overload.

Constructive Solutions #

Make sure resources.requests.cpu blocks are explicitly declared in container manifests before enabling HPAs.

# CORRECT: Deployment configuration with complete requests
spec:
  containers:
  - name: customer-api
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "1000m"
        memory: "512Mi"

Anti-Pattern 3: Setting minReplicas: 1 for Critical Services #

Setting the minReplicas: 1 parameter on HPA specs in production to minimize VM rental cost spending outside busy hours (e.g. at night).

# File: k8s/production-hpa.yaml
# ANTI-PATTERN: minReplicas: 1 at the production level
spec:
  minReplicas: 1 # DON'T: Potential single point of failure!
  maxReplicas: 15

Operational Consequences #

• No High Availability: When HPAs scale down to 1 pod at night, that single pod acts as a single point of failure. If the pod crashes due to Node network errors or OOMKilled, the system totally dies for dozens of seconds until the replacement pod is ready.
• Rolling Update Downtime: When we do application release updates at night with the default maxUnavailable: 25% configuration (or a minimum of 1 pod), Kubernetes is forced to kill the only running pod before starting the new version pod, triggering transaction failures for users during the release process.

Constructive Solutions #

Always set the minimum replica limit (minReplicas) to at least 2 (3 recommended for multi-AZ) on production clusters to guarantee 24/7 service stability.

Anti-Pattern 4: Backups Without Restoration Tests #

Operating cluster etcd backup schedules or daily database snapshots to S3 buckets, then assuming the cluster is safe without ever testing the validity of those backup files.

ANTI-PATTERN: "Backup files pile up in S3, the cluster is guaranteed safe."
  1. Scheduled Velero backups successfully send .tar.gz files to S3.
  2. Database pg_dump runs every day at 1 AM.
  3. However, the team never tests the restoration process of those files.

Operational Consequences #

When a real disaster happens in the cluster (e.g. etcd hacking or volume data damage), the team tries recovering data, but the process totally fails because:

• The backup files turn out corrupted due to unnoticed network transfer failures.
• The database dump file formats are incompatible with the new database engine major version upgraded in the DR cluster.
• The S3 IAM authorization policy has expired so the cluster can’t download backup files.

Constructive Solutions: Periodic Restoration Test Workflows (DR Drills) #

Schedule disaster simulation drills (Disaster Recovery Drills) periodically at least every 3 months. Test the Velero backup file restoration process to isolated test namespaces using namespace mappings:

# 1. Run the restore from the production backup to a special test namespace
$ velero restore create dr-drill-restore-20260617 \
  --from-backup daily-prod-backup-20260617 \
  --namespace-mappings prod-apps:prod-restore-test

# 2. Verify the health status of the restored Pods
$ kubectl get pods -n prod-restore-test

# 3. Verify database data consistency by running test queries
# 4. Delete the test namespace after verification for cost efficiency
$ kubectl delete namespace prod-restore-test

Anti-Pattern 5: Missing Requests/Limits (BestEffort QoS Class) #

Allowing developer teams to submit application manifests to production clusters without restricting CPU and memory allocations at the container level.

The Impact of Host Kernel-Level oom_score_adj Values #

The Kubernetes kubelet translates requests/limits allocations into Linux kernel /proc/[PID]/oom_score_adj configurations internally based on QoS classes:

If our containers enter the BestEffort class due to missing resource allocations, those containers are instantly swept away by the host OOM Killer when the Node suffers high RAM consumption, triggering sudden downtime without any transition time tolerance.

Constructive Solutions #

Use LimitRange objects in every namespace to automatically insert default limits, and apply ResourceQuotas to restrict developer team total consumption.

Anti-Pattern 6: Single-Availability Zone (Single-AZ) Node Pools #

Building production clusters by placing all worker nodes in the same physical Availability Zone (e.g. in GCP: asia-southeast1-a or in AWS: ap-southeast-1a).

Operational Consequences #

If the cloud provider suffers regional electrical failures or planned maintenance on the ap-southeast-1a zone rack, all our worker nodes die simultaneously. Even if we have 10 Pod replicas and configured PDBs, our application still totally dies because there’s no remaining alternative physical infrastructure to run the Pods.

Constructive Solutions #

Design node pools across physical availability zones (minimum 3 AZs, e.g. ap-southeast-1a, ap-southeast-1b, and ap-southeast-1c) and use the topologySpreadConstraints property on Pod manifests to evenly spread replicas.

# CORRECT: Cross-AZ spread configuration in Deployments
spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: core-api

Anti-Pattern 7: Clusters That Are Never Upgraded (Upgrade Debt) #

Letting production clusters run outdated Kubernetes versions declared End-Of-Life (EOL) by the community (e.g. still running v1.22 in 2026).

The Danger of Deprecated APIs #

Kubernetes periodically removes deprecated API versions. For example:

• The extensions/v1beta1 API for Ingress has been removed since v1.22 and replaced by networking.k8s.io/v1.
• The policy/v1beta1 API for PodSecurityPolicies was removed since v1.25.

If we delay upgrades too long (e.g. jumping directly from v1.22 to v1.29), we can’t do instant direct upgrades because the API differences are too far apart. We must do gradual minor version by minor version upgrades, cutting and modifying our application manifests at every stage, triggering very high human configuration error risks during the transition process.

Constructive Solutions #

Schedule periodic cluster version upgrade maintenance at least once or twice a year. Leverage Release Channels features on managed Kubernetes (like the GKE Regular Channel) to automate gradual, staging-tested cluster version evaluation and upgrade processes.

Production Anti-Pattern Prevention Checklist #

Use this checklist as a final audit guide before declaring your cluster ready for large-scale operations:

SERVICE AVAILABILITY HARDENING (HA):
  □ Every critical production application pod has a minimum of 3 active replicas.
  □ PodDisruptionBudget (PDB) objects are configured to limit downtime during node drains.
  □ HPAs are set with a 'minReplicas' value of at least 2 (or 3) to avoid cold-start outages.
  □ 'sleep 10' preStop lifecycle hooks are installed on containers to prevent HTTP 502s during rolling updates.

RESOURCE MANAGEMENT & AUTOSCALING:
  □ All containers have clear requests and limits specs (avoiding BestEffort QoS).
  □ HPAs are only configured on containers with defined 'requests.cpu' specs.
  □ VPAs and HPAs don't monitor the same metrics on one Deployment to prevent 'scale wars'.
  □ LimitRanges are installed in every active namespace to insert automatic default values.

DISASTER RECOVERY & INFRASTRUCTURE:
  □ Disaster simulation drills (DR Drills) are routinely run to isolated namespaces at least every 3 months.
  □ Relational database backups use native dumps (pg_dump) for data consistency.
  □ Backup files are stored encrypted in S3 with the Object Lock (WORM) feature active.
  □ Worker node pools are evenly spread across at least 3 Availability Zones (Multi-AZ).
  □ Cluster Kubernetes minor versions are periodically upgraded and don't lag > 2 minor versions.

Summary #

• PDBs Protect Node Drains — Don’t let infrastructure maintenance activities kill applications; must create PDBs (minAvailable / maxUnavailable) for every production deploy.
• HPAs Need Requests — Always enable requests parameters on container CPU/Memory so HPAs don’t get stuck in the <unknown> status when reading metrics.
• Avoid minReplicas: 1 — Keep the HPA minimum replica at 2 to avoid application deaths when single pods crash or suffer version swaps.
• Test Your Restores Regularly — Remember that S3 backup files have no value if we never train teams to successfully restore them in clusters.
• Turn Off BestEffort QoS — Secure physical Nodes from memory starvation threats by forbidding pods running without declarative resources parameters.
• Use Multi-AZ Node Pools — Protect clusters from regional infrastructure crises by evenly spreading worker node pools across Availability Zones.
• Upgrade Periodically — Prevent upgrade debt accumulation by regularly upgrading Kubernetes minor versions to close cluster security CVE holes.