Storage Anti-Patterns #

In modern microservice application architecture, data is the most valuable asset. We can easily manage stateless applications; if a web container crashes or dies, we just let Kubernetes spin up a fresh replica from a clean image. However, when dealing with stateful workloads like transactional databases, message brokers, or file storage media, our fault tolerance becomes very strict.

Storage is the area where configuration mistakes most often go undetected from the start. Everything might look smooth during the first deployment. But when the cluster experiences hardware failures, node maintenance, or version upgrades, wrong configurations can trigger permanent data loss.

This article comprehensively reviews the storage anti-patterns most commonly encountered in production Kubernetes cluster environments, their bad business consequences, and concrete solution code examples to avoid them.

Anti-Pattern 1: Storing State in the Container Filesystem (Ephemeral Storage) #

This is the most classic and fundamental mistake we often encounter when developer teams newly migrate traditional monolithic applications into Kubernetes. They’re used to writing user upload images, log files, session state, or even SQLite database files directly into the application’s local folder (e.g. /app/data).

The Bad Consequences #

Containers in Kubernetes are designed to be ephemeral. Every time a container crashes, gets restarted by the kubelet, or moves to another node due to routine maintenance (rolling updates), Kubernetes discards the old container along with its entire writable filesystem layers, then creates a new container from a clean base image. As a result, all user upload files or local database transaction data are destroyed forever.

Code Comparison #

Wrong Manifest Code (Writing Directly to the Container Layer) #

# DON'T DO THIS: Writing data without a volume mount
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-without-storage
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-container
          image: my-app:v1.0.0
          # The app writes upload files to /var/www/uploads directly

Solution Code (Using Volumes for Data Persistence) #

# SOLUTION: Separate persistent data with PVCs and temporary data with emptyDir
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-safe-storage
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-container
          image: my-app:v1.0.0
          volumeMounts:
            - name: persistent-uploads-volume
              mountPath: /var/www/uploads
            - name: temporary-cache-volume
              mountPath: /tmp/cache
      volumes:
        - name: persistent-uploads-volume
          persistentVolumeClaim:
            claimName: user-uploads-pvc # Points to an external PVC
        - name: temporary-cache-volume
          emptyDir: {} # Cache may be lost when the Pod dies, but doesn't dirty the container filesystem

Anti-Pattern 2: Using the Delete Reclaim Policy for Production Databases #

By default, many cloud providers configure their default StorageClass with a Delete volume reclaim policy. That means if the PersistentVolumeClaim (PVC) object in Kubernetes is deleted, the physical disk volume on the cloud provider (like AWS EBS or Google Cloud Persistent Disk) is also instantly deleted.

The Bad Consequences #

Humans make mistakes. In an emergency scenario where our database has issues, an engineer might hurriedly delete the database StatefulSet or PVC intending to recreate the manifests. If the StorageClass in use has the Delete policy, as soon as the PVC object is deleted from the API Server, the CSI driver immediately sends a physical disk deletion command to the cloud API. Terabyte-sized production database data can vanish within seconds with no chance to restore from a cloud Recycle Bin.

Code Comparison #

Wrong Manifest Code (Delete Reclaim Policy on a DB StorageClass) #

# DON'T USE THIS FOR IMPORTANT PRODUCTION DATA
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: DB-storage-class
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete # Very dangerous: Delete PVC = Delete Physical Cloud Disk
parameters:
  type: gp3

Solution Code (Applying the Retain Reclaim Policy) #

# SOLUTION: Securing physical volumes with the Retain policy
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: secure-database-sc
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain # Very safe: If the PVC is deleted, the physical cloud volume SURVIVES
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

• With the Retain policy, if the database PVC is accidentally deleted, the physical PersistentVolume (PV) status in the Kubernetes cluster changes to Released (released). The data in the cloud stays safe, and the cluster administrator can reclaim that data by creating a new static PV manifest pointing to the same physical disk ID.

Anti-Pattern 3: Sharing a Block Storage Volume (ReadWriteOnce) Across Many Pods #

Many developers think they can save storage costs by creating one large high-performance SSD PVC, then mounting it to many application Pod replicas simultaneously using a Deployment pattern.

The Bad Consequences #

Most cloud block storage (like AWS EBS or GCP Persistent Disk) only supports the ReadWriteOnce (RWO) access mode. That means the volume can only be exclusively attached to one worker node VM at a time.

If we have a Deployment with 3 Pod replicas running on 3 different physical worker nodes, all referencing the same RWO PVC, only the first Pod successfully runs. The other two Pods stay stuck forever in ContainerCreating or Pending status with a Multi-Attach error for volume message.

Let’s look at this failure scenario visualization through the following diagram:

flowchart TD
    PVC["Shared PVC (ReadWriteOnce)"] -- Mount Successful --> Pod1["Pod A (Node 1)"]
    PVC -. Multi-Attach Error .-> Pod2["Pod B (Node 2)"]
    PVC -. Multi-Attach Error .-> Pod3["Pod C (Node 3)"]
    
    style Pod2 fill:#ffcccc,stroke:#ff0000,stroke-width:2px
    style Pod3 fill:#ffcccc,stroke:#ff0000,stroke-width:2px

Code Comparison #

Wrong Manifest Code (Shared RWO PVC Across All Pod Replicas) #

# DON'T DO THIS: One PVC shared by many Pod replicas
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-shared-rwo-failure
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-server
  template:
    metadata:
      labels:
        app: web-server
    spec:
      containers:
        - name: app-container
          image: nginx:alpine
          volumeMounts:
            - name: shared-data
              mountPath: /usr/share/nginx/html
      volumes:
        - name: shared-data
          persistentVolumeClaim:
            claimName: shared-html-pvc # A single RWO PVC

Solution Code (Using volumeClaimTemplates on a StatefulSet) #

For stateful applications (like databases), we must not use a Deployment. We must use a StatefulSet and leverage the volumeClaimTemplates feature so Kubernetes automatically creates unique PVCs for each running Pod replica (e.g. data-db-0, data-db-1, etc.).

# SOLUTION: Using a StatefulSet with a unique PVC template per Pod
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-db
  namespace: production
spec:
  serviceName: postgres-service
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres-container
          image: postgres:15
          volumeMounts:
            - name: db-data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: db-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: secure-database-sc
        resources:
          requests:
            storage: 50Gi

Anti-Pattern 4: Misusing hostPath for Production Workloads #

The hostPath volume type mounts a physical directory directly from the worker node OS where the Pod runs into the container. Developer teams often find this very practical early in development because it doesn’t require complex CSI driver or StorageClass configuration.

The Bad Consequences #

1. Data Inconsistency: If our Pod crashes and gets rescheduled by the Kubernetes scheduler to another worker node, the Pod mounts a directory from the new worker node that’s either empty or holds different data. Our data becomes fragmented across physical nodes.
2. Total Data Loss: If the worker node suffers hardware failure, all data stored on that node’s local disk vanishes. We lose Kubernetes’ high availability and volume portability features.
3. Serious Security Vulnerabilities: If our container gets exploited by a hacker, the hacker can read or modify all sensitive host OS files (like /etc/shadow or /root/.ssh/) through a hostPath mount running with full access permissions.

Code Comparison #

Wrong Manifest Code (Exposing the Physical Node Filesystem) #

# DON'T DO THIS IN PRODUCTION: Data locked to one physical node
apiVersion: v1
kind: Pod
metadata:
  name: insecure-app-pod
  namespace: production
spec:
  containers:
    - name: app
      image: node:18
      volumeMounts:
        - name: local-storage
          mountPath: /app/data
  volumes:
    - name: local-storage
      hostPath:
        path: /var/lib/my-app-data # Fully dependent on the worker node host filesystem
        type: DirectoryOrCreate

Solution Code (Using Built-in CSI Dynamic PVC Abstractions) #

# SOLUTION: Use a PVC so data volumes can move with the Pod to any node
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-portable-data-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: secure-database-sc
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portable-app-deployment
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: portable-app
  template:
    metadata:
      labels:
        app: portable-app
    spec:
      containers:
        - name: app
          image: node:18
          volumeMounts:
            - name: portable-volume
              mountPath: /app/data
      volumes:
        - name: portable-volume
          persistentVolumeClaim:
            claimName: app-portable-data-pvc

Anti-Pattern 5: Ignoring Ephemeral Storage Limits on Pods #

When we deploy applications, we’re used to limiting CPU and RAM usage (resource requests and limits). However, we often ignore limits for the container’s local disk usage (ephemeral storage). Directories like /tmp or container log files use the worker node’s main system disk capacity where Kubernetes runs.

The Bad Consequences #

If our application has a bug (e.g. writing logs without limits/rotation, or downloading very large temporary files to the /tmp directory), the worker node’s main disk (/var/lib/kubelet) fills up to 100%.

When node disk capacity becomes critical, the local Kubelet triggers DiskPressure status. The Kubelet then takes forced eviction action by randomly killing Pods running on that node to save host OS stability. As a result, our healthy, innocent production application Pods get killed by collateral damage.

Let’s look at this event flow:

flowchart TD
    AppBug["App Writes Unlimited Logs"] -. Fills the disk .-> NodeDisk["Worker Node Disk (100% Full)"]
    NodeDisk -. Triggers status .-> DiskPressure["Kubelet DiskPressure Event"]
    DiskPressure -. Executes .-> Eviction["Force Eviction (Evict Pods)"]
    Eviction -. Kills .-> ProdPods["Other Production Pods (Downtime)"]

Code Comparison #

Wrong Manifest Code (Without Ephemeral Storage Limits) #

# DON'T DO THIS: Prone to filling the worker node disk
apiVersion: v1
kind: Pod
metadata:
  name: risky-app-pod
  namespace: production
spec:
  containers:
    - name: log-writer
      image: alpine
      command: ["/bin/sh", "-c", "while true; do echo 'unlimited logs' >> /var/log/app.log; done"]
      resources:
        limits:
          cpu: "500m"
          memory: "512Mi"
          # ephemeral-storage limits left empty!

Solution Code (Applying Ephemeral Storage Limits) #

# SOLUTION: Limit container disk capacity so it doesn't disturb neighbors
apiVersion: v1
kind: Pod
metadata:
  name: safe-app-pod
  namespace: production
spec:
  containers:
    - name: safe-log-writer
      image: alpine
      command: ["/bin/sh", "-c", "while true; do echo 'safe logs' >> /var/log/app.log; done"]
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
          ephemeral-storage: "500Mi" # Initial disk capacity request
        limits:
          cpu: "500m"
          memory: "512Mi"
          ephemeral-storage: "2Gi" # Maximum container disk usage limit

• If the safe-app-pod Pod writes ephemeral data beyond the 2Gi limit, the Kubelet immediately kills this Pod exclusively without sacrificing the physical worker node’s stability or other production Pods.

Anti-Pattern 6: Not Explicitly Setting storageClassName on PVCs #

When creating PVC manifests, developers often leave the storageClassName property empty, assuming Kubernetes will always pick the cluster’s built-in StorageClass configured by the SysOps team.

The Bad Consequences #

Relying on implicit values in production is a high-risk action.

1. Sudden Performance Degradation: If the platform team migrates or upgrades the cluster and accidentally changes the cluster default StorageClass from a super-fast SSD type (premium-gp3) to a cheap slow HDD (standard-hdd), our newly created PVCs automatically use HDD. As a result, our database suffers drastic IOPS degradation without any application code changes.
2. Total Deployment Failure: If the new cluster has no default StorageClass configured at all, our PVCs stay in Pending status forever, confusing the deployment team because there’s no clear error message in the application manifests.

Code Comparison #

Wrong Manifest Code (Leaving storageClassName Empty) #

# DON'T DO THIS: PVC behavior depends on out-of-cluster settings
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-database-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  # storageClassName omitted!

Solution Code (Writing storageClassName Explicitly) #

# SOLUTION: Declaratively specify the storage class we want
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-database-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: secure-database-sc # Always explicit to guarantee performance
  resources:
    requests:
      storage: 100Gi

Conclusion & Cluster Storage Security Audit Checklist #

Managing stateful workloads in Kubernetes demands high caution. We must treat data as an important asset and design our storage infrastructure anticipating every worst-case possibility (fail-safe design).

Before releasing applications to production, let’s use the following storage audit checklist as a guide to evaluate our cluster:

Summary #

• Must mount volumes for stateful data: Don’t let data be written directly to the container filesystem because it vanishes when the container restarts. Use PVCs for persistent files and emptyDir for caches.
• Retain policy protects data from human error: Set the production database StorageClass reclaim policy to Retain so physical cloud data doesn’t get destroyed if a PVC is accidentally deleted.
• Avoid shared ReadWriteOnce PVCs on Deployments: Block storage can only bind to one worker node. Use a StatefulSet with volumeClaimTemplates to allocate unique PVCs per Pod.
• Leave hostPath out of production: hostPath breaks Pod portability and opens fatal host OS security gaps. Always rely on standard CSI drivers.
• Limit container ephemeral-storage: Prevent DiskPressure incidents on worker nodes caused by container log/data leaks by applying ephemeral storage limits in Pod manifests.
• Be explicit about storageClassName: Always specify the target StorageClass in writing in our PVC manifests to prevent unexpected performance changes during cluster migrations.

← Previous: Dynamic Provisioning   Next: Storage Performance →