Ephemeral vs Persistent Storage #

In the Kubernetes ecosystem, separating application logic (compute) from data storage (state) is a fundamental design principle distinguishing modern architecture from traditional server models. When we deploy applications in Kubernetes, we no longer treat servers (worker nodes) as static entities storing our files forever. Instead, worker nodes and the Pods on them can be shut down, moved, or destroyed at any time by the scheduler.

Therefore, understanding how data is stored in the cluster becomes crucial. Kubernetes divides the storage world into two main categories with opposing operational philosophies: Ephemeral Storage and Persistent Storage. Choosing the wrong approach for our workload type can cause catastrophic data loss on one side, or unnecessary cost waste and performance overhead on the other. This article builds a mature architectural mental model of both storage categories before we dive into the technical details of PV, PVC, and StorageClass.

The Basic Nature of the Container Filesystem: The Immutable Writable Layer #

Before discussing the difference between ephemeral and persistent volumes, we must understand how Linux containers manage their filesystems at the basic level.

When a container runtime (like containerd) runs a container from an application image (e.g. nginx:latest), the container doesn’t run on a directly modifiable filesystem. Containers use layered filesystem technology like OverlayFS or UnionFS:

  1. Read-Only Image Layers: The layers composing the container image are static and unchangeable (read-only).
  2. Thin Writable Layer: The container inserts a thin layer above the image layers that can be written to (writable layer). All new file creation, log file modification, or temporary upload storage operations are written to this writable layer.
[Container Filesystem Structure (OverlayFS)]
  │
  ├──► Writable Layer (Thin & Temporary) ➔ Written while the app runs
  │                                        (Deleted if the container crashes/restarts!)
  │
  └──► Read-Only Image Layers (Static)    ➔ App binaries, base OS, libraries

[!WARNING] This writable layer’s lifecycle is absolutely tied to the container’s physical lifecycle. If our main container crashes from memory exhaustion (OOMKilled) and the Kubelet triggers a new container restart process, the old writable layer is completely destroyed. The new container starts with a fresh, clean writable layer, and all data the old container wrote to its local filesystem is lost beyond recovery.

This is a deliberate design decision (immutability) to ensure stateless applications can be replicated in parallel without carrying dirty local state. But what if our application needs to store data that must survive restarts, or share data between containers? That’s where the Kubernetes Volume abstraction comes in.

Storage Lifecycle Flow: Ephemeral vs Persistent #

To clarify the lifecycle difference between ephemeral and persistent volumes, let’s look at the execution flow visualization when a Pod crashes or moves nodes:

flowchart TD
    subgraph Ephemeral["Ephemeral Volume (emptyDir)"]
        E_PodStart["Pod Runs on Node-1"] --> E_DirCreate["Kubelet Creates an Empty Directory on Node-1"]
        E_DirCreate --> E_Write["Container Writes Data to /data"]
        E_Write --> E_PodCrash["Pod Crashes / Is Shut Down"]
        E_PodCrash --> E_DirDelete["Kubelet Deletes the Empty Directory & Its Contents"]
        E_DirDelete --> E_Lost["DATA LOST"]
    end

    subgraph Persistent["Persistent Volume (PV/PVC)"]
        P_PodStart["Pod Runs on Node-1"] --> P_Claim["Pod Requests a PVC"]
        P_Claim --> P_Bind["PVC Bound to a PV (Cloud/NFS Disk)"]
        P_Bind --> P_Write["Container Writes Data to /data (External Disk)"]
        P_Write --> P_PodCrash["Pod Crashes / Moves to Node-2"]
        P_PodCrash --> P_Unmount["Volume Unmounted from Node-1"]
        P_Unmount --> P_Mount["Volume Mounted to Node-2"]
        P_Mount --> P_Recover["New Pod on Node-2 Reads the Old Data"]
        P_Recover --> P_Safe["DATA STAYS SAFE"]
    end

Ephemeral Storage: Temporary Storage Tied to the Pod #

Ephemeral volumes are storage whose lifecycle is directly tied to the Pod’s lifecycle. As long as the Pod runs on the worker node, the data in these volumes stays safe. However, when the Pod object is declaratively deleted (e.g. due to Deployment scale-down or node evacuation), all data in the ephemeral volume is permanently deleted.

Kubernetes provides several commonly used built-in ephemeral volume types:

1. emptyDir: An Empty Directory on Disk or RAM #

emptyDir is the simplest and most commonly used ephemeral volume type. When a Pod is scheduled on a worker node, the Kubelet creates an empty directory in the worker node’s local filesystem. All containers in the same Pod can read and write files to this directory simultaneously.

Here’s a production-grade emptyDir usage manifest:

apiVersion: v1
kind: Pod
metadata:
  name: cache-worker-pod
spec:
  volumes:
  - name: disk-cache
    emptyDir: {}                     # Stores data on the worker node's HDD/SSD (local)
  - name: memory-cache
    emptyDir:
      medium: Memory                 # Stores data in RAM (tmpfs) — Very Fast!
      sizeLimit: 512Mi               # Must limit the memory size
  containers:
  - name: app-runner
    image: company/app-image:v1
    volumeMounts:
    - name: disk-cache
      mountPath: /app/tmp-disk
    - name: memory-cache
      mountPath: /app/tmp-ram

Using Memory-Based emptyDir (RAM) #

By setting medium: Memory, Kubernetes creates a tmpfs (virtual RAM) filesystem on the worker node.

  • Advantage: Very high read/write speed because I/O operations happen directly in physical memory without touching mechanical disks/SSDs.
  • Risk: Data in tmpfs cuts into the container’s cgroups memory allotment. If we don’t include sizeLimit and the container writes files beyond the node’s physical memory, the worker node experiences an OOM (Out of Memory) failure.

Valid emptyDir Use Cases: #

  • Temporary caching layers that don’t damage the main application’s function if lost.
  • Shared workspaces between containers in one Pod (e.g. an init container downloads asset files, and the main container serves them).
  • Temporary video rendering or file compression processes whose results are immediately uploaded to Object Storage (like S3 or GCS).

2. ConfigMap and Secret as Volumes #

ConfigMaps and Secrets are also categorized as ephemeral volumes. Kubernetes presents configuration data or sensitive credentials as static text files mounted into the container directory. If the Pod is destroyed, the mount point disappears, but the original ConfigMap and Secret data stays safely stored in the control plane’s etcd database.

For stateful workloads — like databases, queue clusters, or shared file systems — we need storage that doesn’t care whether our Pod dies, restarts, or moves to another physical node at the far end of the data center. Our data must survive and be transparently accessible again by the replacement Pod.

In Kubernetes, we break the tight link between compute (Pod) and storage lifecycles using a storage trinity of components: StorageClass, PersistentVolume (PV), and PersistentVolumeClaim (PVC).

[The Kubernetes Storage Trinity]

  StorageClass          ➔ The "recipe" template for requesting storage from a cloud provider.
                           (Examples: SSD type, encryption enabled, multi-zone).

  PersistentVolume      ➔ The physical representation of storage already provisioned in the backend.
                           (Examples: AWS EBS 100GB volume, GCP PD 50GB, NFS share).

  PersistentVolumeClaim ➔ The storage request ticket submitted by application developers.
                           (Example: "I want a 10GB disk with ReadWriteOnce access").

This mechanism applies the separation of concerns principle:

  • Platform Engineer/System Administrator: Configures StorageClass and sets up driver integrations (CSI - Container Storage Interface) to infrastructure providers (AWS, GCP, VMware, or Ceph).
  • Application Developer: Just writes a PersistentVolumeClaim manifest requesting a certain storage capacity. Developers don’t need to know the storage hardware brand used behind the scenes or the cloud provider API calls to create a new disk.

Understanding Access Patterns: Access Modes #

When we submit a storage request with a PVC, we must declare the Access Mode, which determines how the volume will be mounted across all worker nodes in the cluster. Choosing the wrong access mode can leave Pods stuck in ContainerCreating status forever due to physical hardware restrictions.

Here are the 4 Access Modes available in Kubernetes:

1. ReadWriteOnce (RWO) #

The volume can only be mounted as read-write by one single worker node at a time.

  • Behavior: If Pod A bound to an RWO volume runs on node-1, then Pod B running on node-2 can’t read or write to that volume. However, if two Pods run on the same worker node (node-1), some cloud providers allow both to attach to the same RWO volume (although this isn’t recommended).
  • Use Case: Transactional databases (PostgreSQL, MySQL, Redis) needing exclusive I/O to prevent data corruption.

2. ReadOnlyMany (ROX) #

The volume can be mounted as read-only by many worker nodes simultaneously.

  • Use Case: Static file distribution, shared source code repositories, or static image asset directories read by many parallel web server Pods without needing application writes.

3. ReadWriteMany (RWX) #

The volume can be mounted as read-write by many worker nodes simultaneously.

  • Behavior: All Pods spread across different worker nodes can read and write to the same folder concurrently.
  • Backend Technology: This mode is not supported by traditional cloud block storage types like AWS EBS or GCP Persistent Disk. RWX requires distributed file-based storage like NFS, AWS EFS, GCP Filestore, CephFS, or GlusterFS.
  • Use Case: CMS systems (like WordPress) needing a shared image upload directory across Pod replicas, or centralized batch log processing.

4. ReadWriteOncePod (RWOP - Kubernetes v1.22+) #

The volume can only be mounted as read-write by one single Pod across the whole cluster.

  • Behavior: This is stricter protection than RWO. If Pod A has mounted an RWOP volume, no other Pod (even on the same worker node) may touch that volume.
  • Use Case: Guarantees absolute data safety for critical stateful applications against accidental concurrent access.

Generic Ephemeral Volumes: The Hybrid Approach (Semi-Persistent) #

Modern Kubernetes introduces an advanced feature called Generic Ephemeral Volumes. This feature combines the easy lifecycle of ephemeral volumes with the power of persistent volume features.

With Generic Ephemeral Volumes, we can ask Kubernetes to dynamically create a persistent storage volume (e.g. a high-performance Cloud SSD) when the Pod is created, and Kubernetes will automatically delete that persistent volume when the Pod is destroyed.

Here’s a Generic Ephemeral Volume usage manifest:

apiVersion: v1
kind: Pod
metadata:
  name: batch-analyzer-pod
spec:
  containers:
  - name: analyzer
    image: company/analyzer:v1.0
    volumeMounts:
    - name: scratch-storage
      mountPath: /data/scratch
  volumes:
  - name: scratch-storage
    ephemeral:                               # Marks the hybrid ephemeral volume
      volumeClaimTemplate:
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "premium-ssd-sc" # Dynamically creates a physical SSD
          resources:
            requests:
              storage: 50Gi

Why Do We Need Generic Ephemeral Volumes? #

  • Large Capacity: Regular emptyDir volumes eat the worker node’s root disk space. If we need a 50GB disk just for a batch job’s temporary workspace, using emptyDir risks filling the node disk (disk pressure).
  • Rich Storage Features: We can use special StorageClass features, like cloud disk encryption, high IOPS, or SSD types, which a plain local node empty directory can’t provide.
  • Clean Lifecycle: We don’t need to manually clean up PVCs after our batch job finishes; Kubernetes deletes them automatically.

Production Considerations: Managed Services vs In-Cluster Storage #

Before deciding to deploy all our stateful applications inside Kubernetes using Persistent Volumes, we must be operationally realistic. Running state inside Kubernetes requires high-level expertise (day-2 operations overhead).

Here’s an architectural decision guide we can use:

1. When to Use a Managed Database Service (Bypass Kubernetes)? #

For main transactional relational databases with high business SLAs (like financial databases or primary user data), consider not deploying them inside Kubernetes.

  • Options: Use cloud provider managed services like AWS RDS, GCP Cloud SQL, or Azure Database.
  • Reason: Managed services provide automatic backups, multi-zone replication with automatic failover, and OS patching fully managed by the provider. We don’t need to worry about PV disk filesystem corruption or StatefulSet quorum coordination when worker nodes die.

2. When to Run Databases Inside Kubernetes (PV/PVC)? #

  • Non-relational databases with built-in distributed architecture (shared-nothing architecture), like Elasticsearch, Cassandra, or Apache Kafka clusters, which are designed to be resilient against random node death.
  • Testing environments (staging / development) where we want to cut costs by not renting expensive managed cloud databases.
  • Stateful applications packaged with official Operator Patterns (e.g. the Zalando Postgres Operator or Confluent Kafka Operator) that automate failover processes inside the Kubernetes cluster.

3. Use Object Storage (S3/GCS) for Static Files #

For static file storage like user profile images, PDF documents, or web assets, avoid using ReadWriteMany (RWX) persistent volumes because cloud NFS/EFS performance is usually slow for small file operations.

  • Solution: Use the Object Storage SDK (AWS S3, Google Cloud Storage) directly in our application code. It’s much cheaper, infinitely scalable, and has more stable performance.

Storage Management Anti-Patterns & Their Solutions #

Here are three fatal mistakes that most often destroy data or disrupt production cluster performance:

Anti-Pattern 1: Storing Application Log Files Directly on the Container Writable Layer #

Letting the application write log files (like Nginx access logs or Java debug logs) directly into the container’s internal filesystem without mounting a volume.

ANTI-PATTERN: Writing Logs to /var/log/app.log Inside the Container Writable Layer
// WHAT WE DO:
- Run an application container constantly dumping log lines to a local container file.

// THE CONSEQUENCES IN PRODUCTION:
- Node Disk Leak: Log files in the writable layer are stored in the worker node's Docker/Kubelet directory.
  If the file grows to 20GB, the worker node's root disk fills up.
- Eviction Loop: The Kubelet detects disk pressure (DiskPressure) on the node and starts
  force-evicting Pods from that node to free disk space.
- Lost Audit Evidence: Once the container restarts, all historical logs are completely gone,
  making security incident or application bug analysis harder.

✓ THE RIGHT SOLUTION:
- Configure the application to always dump logs to the standard output channels (stdout/stderr).
- Let cluster-level log collector daemons (like Fluent Bit or Loki Promtail) running
  as DaemonSets capture that stdout log and ship it to a central log server (Elasticsearch/Loki).

Anti-Pattern 2: Assuming ReadWriteOnce (RWO) Volumes Can Be Shared Across Nodes in a Deployment #

Deploying a 3-replica web application (Deployment) attached to a single ReadWriteOnce PVC.

ANTI-PATTERN: spec.replicas: 3 Bound to a Single RWO PVC
// WHAT WE DO:
- Create a single PVC object named `web-pvc` with `ReadWriteOnce` mode.
- Create a Deployment with `replicas: 3`, all pointing to the same `web-pvc`.

// THE CONSEQUENCES IN PRODUCTION:
- Multi-Attach Error: The kube-scheduler schedules the 3 replica Pods to different worker nodes
  (e.g. `node-1`, `node-2`, `node-3`).
- The first Pod on `node-1` runs successfully because it manages to mount the volume.
- The second and third Pods on `node-2` and `node-3` stay stuck forever in `ContainerCreating` status
  with the error `Multi-Attach error for volume: volume-xyz can only be attached to single node`.

✓ THE RIGHT SOLUTION:
- If our application truly must share one common write folder across nodes,
  use the `ReadWriteMany` (RWX) access mode and make sure our StorageClass supports NFS/EFS.
- If the application is a stateful database needing separate storage per Pod,
  don't use a regular Deployment. Use a StatefulSet with the `volumeClaimTemplates` property
  to automatically create unique PVCs for each Pod replica (`db-0`, `db-1`, `db-2`).

Anti-Pattern 3: Using Memory-Based emptyDir Without a Maximum Limit (sizeLimit) #

Creating a RAM-type (Memory) cache volume chasing instant performance without limiting the writable data size.

ANTI-PATTERN: emptyDir: { medium: Memory } Without sizeLimit
// WHAT WE DO:
- Create a tmpfs (RAM) volume to hold web session caches.
- Forget to limit the maximum memory size in the YAML manifest.

// THE CONSEQUENCES IN PRODUCTION:
- Node RAM Monopoly: If traffic spikes, the application keeps writing session data to the tmpfs folder.
  Because tmpfs uses the worker node's physical RAM, the node's RAM gets drained completely.
- Node Crash: The Linux kernel detects critical memory exhaustion on the physical host node.
  Important system processes (including the kubelet or docker daemon) can be randomly killed by the OOM-killer,
  causing the worker node to die completely (*Kernel Panic* or *NotReady*).

✓ THE RIGHT SOLUTION:
- Always declare the `sizeLimit` property when using Memory-based emptyDir:
  emptyDir:
    medium: Memory
    sizeLimit: "256Mi"
- Make sure this sizeLimit is aligned with the container's memory request & limit so it doesn't trigger
  OOMKilled on our own container.

Summary #

  • Immutable Writable Layer — By default, the container’s local filesystem is transient; its data is destroyed when the container crashes or restarts.
  • Ephemeral Volumes (emptyDir) — Live and die with the Pod lifecycle; perfect for caches, data processing scratch space, or exchanging files between containers in one Pod.
  • Tmpfs for Performance — Use medium: Memory-based emptyDir for high-speed file processing, but must limit its size with the sizeLimit property.
  • Persistent Volumes (PV/PVC) — Store data independently outside the Pod lifecycle; data stays safe even if the Pod is deleted or moves to another physical worker node.
  • Access Modes — Specify access precisely: ReadWriteOnce (RWO) for single worker node locking (db), and ReadWriteMany (RWX) for shared folders across nodes (NFS).
  • Generic Ephemeral Volumes — Leverage this hybrid pattern to request large dynamic cloud SSDs automatically deleted when the Pod finishes working.
  • Avoid the Writable Layer for Logs — Don’t write log files to the container’s internal disk; use stdout so logs can be centrally collected without filling the node disk.
  • Use Managed DBs for Credibility — For main production databases, use cloud provider managed services (RDS/Cloud SQL) to minimize data loss risk.

← Previous: QoS Class   Next: Volume →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact