Storage Problems in Distributed Systems #

In modern software engineering, we’re often captivated by the ease Kubernetes offers in managing stateless applications. We can easily replicate, autoscale, and rolling-update without worrying about losing application internal state. However, when we start stepping into the world of stateful workloads — like PostgreSQL databases, Kafka clusters, Redis caches, or Elasticsearch — we face a bitter reality.

Storing data in distributed systems isn’t just writing files to a local disk. When applications run across dozens of physical worker nodes that can suffer power outages, network disruptions, or force evictions at any time by the scheduler, data storage becomes the most complex area of the entire cluster operation. This article dissects the unique challenges of storage in distributed systems, why split-brain is the biggest database disaster, the cloud disk detachment handling flow, and a mature architectural decision formula.

Problem 1: Pod Mobility vs Data Immobility (Data Locality vs Portability) #

In Kubernetes, Pods are designed for high mobility. If the worker node hosting a Pod suffers hardware failure, the control plane detects NotReady status and the scheduler immediately moves that Pod to another healthy worker node.

However, physical data can’t move as fast as Pods. If the data lives on worker node A’s local disk, it’s physically locked there. A Pod moved to worker node B starts with an empty directory, losing all its historical state.

To overcome local data immobility, we use external storage solutions (like cloud persistent disks or network storage). When a Pod moves, that external storage volume must be detached from the old node and reattached to the new one.

Here’s a visualization of the volume detach/attach process during a worker node failure:

flowchart TD
    subgraph Node1["Worker Node 1 (NotReady)"]
        N1_Crash["Node-1 Crash / Power Outage"] --> N1_Stuck["Cloud Disk 'vol-xyz' Stays Locked (Attached)"]
    end

    subgraph ControlPlane["Control Plane (API & Controller)"]
        N1_Stuck --> DetachDelay["Detect NotReady -. Wait 6 Minutes (Timeout) .-> Force Detach"]
        DetachDelay --> ForceDetach["Send Force Detach Command to the Cloud API"]
    end

    subgraph Node2["Worker Node 2 (Healthy)"]
        ForceDetach --> AttachNew["Attach Disk 'vol-xyz' to Node-2"]
        AttachNew --> MountVol["Mount the Volume to the New Pod"]
        MountVol --> PodReady["PostgreSQL Pod Back to Running & Data Safe"]
    end

The Detach/Attach Timeout Problem #

Although the diagram above looks smooth, in the real world this process is often hampered by race conditions and cloud provider API latency.

1. Stuck Volume: When node-1 dies suddenly, the Cloud Provider API still records that disk vol-xyz is exclusively attached to node-1.
2. Safety Lock: The cloud provider refuses to attach the disk to node-2 because it fears node-1 might actually still be running and writing data simultaneously (data corruption prevention).
3. Timeout Delay: Kubernetes must wait for the built-in timeout (usually around 5 to 6 minutes) before forcing a force detach command through the Cloud API. During this wait, our database Pod stays stuck in ContainerCreating or Pending status with the error: Volume is already exclusively attached to one node.

Problem 2: Concurrent Access Patterns and Network I/O Bottlenecks #

When we deploy a web application with many replicas (Deployment), we often want all those Pod replicas to read and write files to the same folder (e.g. an image upload directory).

Block Storage vs File Storage #

Physical hardware storage constraints limit how data can be accessed concurrently:

flowchart TD
    subgraph Block["Block Storage (RWO / RWOP)"]
        B_Pod1["Pod-1 (Node-A)"] -->|"Direct Disk Sector Access<br/>(Very Fast)"| B_Disk["Cloud Disk"]
        B_Pod2["Pod-2 (Node-B)"] --x|"Rejected by the Cloud Provider!"| B_Disk
    end

    subgraph File["File Storage (RWX)"]
        F_Pod1["Pod-1 (Node-A)"] --> F_Server["NFS / EFS Server"]
        F_Pod2["Pod-2 (Node-B)"] --> F_Server
        F_Server -->|"Operations Through Network Protocol<br/>(Has Latency)"| F_Sys["File System"]
    end

1. Block Storage (AWS EBS, GCP PD):
   • Provides a raw block device directly to the node OS.
   • Only supports ReadWriteOnce (RWO) mode because standard filesystems (like ext4 or xfs) aren’t designed to be mounted on multiple machines simultaneously without low-level metadata coordination (distributed lock manager). If forced, the filesystem instantly corrupts.
   • Performance: Very fast because access latency approaches local disk.
2. File Storage (NFS, Amazon EFS, Azure Files):
   • Provides a ready-to-use filesystem accessed over network protocols (NFS or SMB).
   • Supports ReadWriteMany (RWX) mode, so hundreds of Pods across worker nodes can write files simultaneously.
   • Performance: Has significant network latency overhead. Every file operation (creating files, scanning directories, or writing log lines) must go through TCP/IP networking. If our application constantly performs thousands of small file read/write operations, application performance drops drastically (I/O bottleneck).

Problem 3: The Biggest Distributed Database Danger: Split-Brain #

When we run a database with distributed cluster topology (like PostgreSQL Master-Slave, Elasticsearch, or MongoDB Replica Sets) in Kubernetes, we rely on network consensus protocols to determine which node acts as Primary (writes) and which nodes act as Secondary (reads).

The Network Partition Scenario #

Imagine a 3-node database cluster split into two different network zones due to a network switch failure:

flowchart LR
    subgraph ZoneA["Network Zone A"]
        db0["db-0 (Primary)<br/>(Isolated)<br/>Can only see itself"]
    end

    subgraph ZoneB["Network Zone B"]
        db1["db-1 (Slave)"]
        db2["db-2 (Slave)"]
        db1 --- db2
        noteB["Two nodes can see each other<br/>and form quorum (2 of 3)"]
    end

    ZoneA -. "NETWORK PARTITION" .-> ZoneB

If this partition happens:

1. The Right Side (Zone B): Nodes db-1 and db-2 detect they can’t reach Master db-0. Because the two of them form a majority quorum (2 of the total 3 nodes), they automatically promote db-1 as the new Master. The right side now accepts write transactions from applications.
2. The Left Side (Zone A): The isolated db-0 node might still be running and assume it’s still the valid Master if applications in zone A can still reach it. It keeps accepting write transactions.
3. Split-Brain Occurs: Now we have two active Masters writing different data to their own disks without any synchronization. Once the network partition heals, the data on both sides has diverged (diverged data). Manually reconciling this conflicting data is the biggest nightmare of every database team.

Consensus Protection and Fencing Solutions: #

To avoid split-brain, modern database clusters use consensus algorithms like Raft or Paxos:

• The Master node must constantly verify it still has majority quorum support. If db-0 detects it’s isolated (only 1 active node), it must immediately self-demote to read-only automatically (self-demotion).
• CSI Fencing / STONITH (Shoot The Other Node In The Head): The Kubernetes Controller system force-cuts the Persistent Volume access from Pod db-0 through the Cloud Provider API before promoting db-1 as the new master. This guarantees the old master can never write a single byte of data to the physical disk again.

Problem 4: Performance Degradation from Noisy Neighbors #

In multi-tenant Kubernetes environments (where many developer teams share the same cluster), our storage volumes are often placed on the same shared storage backend pool (shared storage pool).

If team A runs a batch processing application aggressively processing 100GB of raw data, the storage network I/O bandwidth gets fully drained. As a result, team B running a low-latency transactional database on a worker node connected to the same storage backend suffers severe performance degradation (noisy neighbor effect). SQL query write latency that normally completes in 3ms balloons to 80ms, triggering transaction failures and system timeouts.

Noisy Neighbor Prevention Strategies: #

• IOPS & Throughput Limits: Always configure I/O limits at the StorageClass level if our cloud provider supports it (e.g. setting iops and throughput parameters on AWS gp3 disks).
• Dedicated Storage Node Pools: Use node labels, taints, and tolerations to deploy I/O-intensive applications on dedicated node pools connected to isolated storage networks.

Problem 5: Day-2 Operation Challenges (Resizing & Consistent Backups) #

Running storage in distributed systems brings complex long-term maintenance challenges.

1. Online Volume Expansion #

When our database disk capacity approaches 90% full, we must enlarge the volume. Kubernetes supports the Online Volume Expansion feature, letting us declaratively change PVC sizes without shutting down Pods.

• How does it work? We change the spec.resources.requests.storage: 100Gi property on the PVC to 200Gi.
• Challenge: The CSI Driver must send resize instructions to the cloud API, then the Kubelet on the worker node must run Linux resize2fs or xfs_growfs commands to expand the running container’s filesystem. If the filesystem crashes mid-way, our database is threatened with corruption.

2. Consistent Backups (Application-Consistent Backups) #

Taking a raw cloud storage volume snapshot (crash-consistent backup) while the database is processing heavy transactions is very dangerous. The snapshot risks capturing data still held in OS memory caches or database buffers not yet written to disk.

• Solution: The backup process must be Application-Consistent. Backup tools (like Velero) must first send instructions to freeze database writes (quiesce/freeze), take the cloud disk snapshot, then unfreeze the database write process.

Architectural Decision Framework: Managed DB vs Self-Managed #

Weighing all the complexity above, we need clear criteria for deciding when it’s worth running a database inside Kubernetes and when to avoid it.

flowchart TD
    Start["Start Stateful Workload Analysis"] --> Q1["Is this data highly business-critical?"]
    Q1 -->|Yes| A1["Use a Managed Service (AWS RDS / GCP Cloud SQL)"]
    Q1 -->|No| Q2["Does your team have Kubernetes experts?"]
    Q2 -->|Yes| A2["Run it in K8s (Use StatefulSet & Official Operator SDK)"]
    Q2 -->|No| A3["Use a Managed DB or Cloud Storage"]

Architectural Comparison Table: #

Distributed Storage Anti-Patterns & Their Solutions #

Here are two fatal mistakes that often paralyze Kubernetes clusters due to mishandled distributed storage:

Anti-Pattern 1: Ignoring volumeBindingMode: WaitForFirstConsumer on Multi-Zone StorageClasses #

Leaving a StorageClass on the default volumeBindingMode: Immediate mode in a Kubernetes cluster spread across multiple Availability Zones (AZs).

ANTI-PATTERN: volumeBindingMode: Immediate on a Multi-AZ Cluster
// WHAT WE DO:
- Create a StorageClass for cloud disks with `Immediate` mode.
- Create a new PVC requesting a 10GB disk.

// THE CONSEQUENCES IN PRODUCTION:
- As soon as the PVC is created, the StorageClass immediately orders the Cloud API to create the physical disk.
  Because the Pod isn't scheduled yet, the StorageClass picks a zone randomly (e.g. Zone A).
- The physical disk is created in Zone A.
- Next, the scheduler processes our Pod. However, because worker nodes in Zone A are full
  or tainted, the scheduler decides to place the Pod on a Zone B worker node.
- The Pod in Zone B tries to attach the physical disk located in Zone A.
- The Pod stays stuck forever in `ContainerCreating` status with the error:
  `Volume node affinity conflict: volume can only be attached to nodes in Zone A, but Pod was scheduled to Zone B`.

✓ THE RIGHT SOLUTION:
- Always set `volumeBindingMode: WaitForFirstConsumer` on all production StorageClasses.
- This mode delays physical disk creation until the scheduler finishes determining which worker node
  will run the Pod. The physical disk is guaranteed to be created in the same Zone as the chosen worker node.

Anti-Pattern 2: Force-Deleting PV/PVCs Stuck in Terminating Status #

Using the --force flag to delete PV or PVC objects from etcd without checking the physical detachment status in the storage backend.

ANTI-PATTERN: kubectl delete pvc mysql-pvc --grace-period=0 --force
// WHAT WE DO:
- Delete the mysql-pvc PVC, but the process gets stuck in `Terminating` status because a Pod is still mounting it.
- Being impatient, we force the deletion with the `--force --grace-period=0` parameters.

// THE CONSEQUENCES IN PRODUCTION:
- Orphaned Storage: Kubernetes instantly deletes the PVC metadata from the etcd database.
- However, because communication with the Cloud API was cut or delayed, the physical disk on the cloud provider
  IS NOT DELETED. That physical disk stays silently active and keeps charging our cloud bill.
- Disk Quota Leak: In the long term, hundreds of these orphaned disks pile up in the cloud,
  exhausting our cloud provider disk creation quota when needed later.

✓ THE RIGHT SOLUTION:
- Never force-delete a PVC stuck in Terminating status.
- Find out why the PVC is stuck by checking whether any active Pod is still mounting it:
  `kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName=="mysql-pvc") | .metadata.name'`
- Stop that Pod first, and Kubernetes will delete the PVC cleanly and automatically.

Summary #

• Physical Data Immobility — Data on disks can’t move as fast as Pods; use cloud persistent disks supporting automatic detach and attach across nodes.
• RWO vs RWX Access Patterns — Block storage (RWO) is very fast but for one node only; shared storage (RWX) supports multi-node but carries network latency overhead.
• The Split-Brain Danger — This is the biggest distributed database risk; use consensus algorithms (Raft/Paxos) and CSI fencing locks to prevent double-writes.
• Detach-Attach Delays — Understand the ~6 minute built-in timeout when moving persistent volumes from a dead worker node to avoid filesystem corruption.
• Noisy Neighbor Mitigation — Protect database performance by applying IOPS limits at the StorageClass level and separating node pools for I/O-intensive workloads.
• Application-Consistent Backups — Freeze database writes before taking volume snapshots so backup data isn’t corrupted when restored.
• Standardize WaitForFirstConsumer — Always use volumeBindingMode: WaitForFirstConsumer on multi-zone StorageClasses so disks are created in the same zone as the Pod.
• Use Managed Services — Consider managed cloud databases (RDS/Cloud SQL) for critical business data to avoid the complexity of managing state in Kubernetes.

← Previous: Volume   Next: PersistentVolume →