StatefulSet + PVC #

In distributed system architecture, running stateful workloads — like PostgreSQL databases, MySQL Galera clusters, Apache Kafka, or Zookeeper — requires double stability guarantees. We need not only network identity and Pod name stability, but also a stable relationship between each Pod replica and its physical data storage medium. If a database Pod dies and gets rebuilt by the control plane, it must be guaranteed to reconnect to the exact same storage volume it used before.

To deliver this double guarantee, Kubernetes tightly synergizes the StatefulSet controller with PersistentVolumeClaims (PVCs). This synergy is dynamically configured through the volumeClaimTemplates mechanism. This article dissects how StatefulSets pair with PVCs, storage lifecycle behavior during database cluster scaling, tactics for modifying immutable StatefulSet parameters using orphan deletion, detailed data migration steps between StorageClasses, and solutions to filesystem permission issues (permission denied).

The StatefulSet-PVC Relationship: A Deterministic Naming Scheme #

When we create a StatefulSet, we’re forbidden from statically declaring PVCs inside the spec.template.spec.volumes block. If we do, all created Pod replicas mount the exact same PVC object, which ends in mount failure if the volume is block storage (RWO).

Instead, we use the volumeClaimTemplates block. This property acts as an automatic PVC factory integrated directly with each Pod’s startup process. Every time the StatefulSet creates a new Pod with ordinal index $i$, the StatefulSet controller in parallel creates a new PVC object in the API Server with deterministic naming:

$$\text{PVC Name Format} = \text{{template-name}}-\text{{statefulset-name}}-\text{{ordinal-index}}$$

To understand how the Kubelet maintains this relationship consistency during crashes or node moves, let’s look at the PVC lifecycle diagram during a scale-down and scale-up cycle:

flowchart TD
    subgraph ScaleDown["Scale-Down Cycle (Replicas: 3 ➔ 1)"]
        SD_Start["Scale-Down Begins"] --> SD_Pod2["1. Delete Pod mariadb-2"]
        SD_Pod2 --> SD_PVC2{"Does the PVC\ndata-mariadb-2\nget deleted too?"}
        SD_PVC2 -- "NO (Intentionally)" --> SD_Retain2["PVC Survives (Bound) & Data Safe"]
        
        SD_Retain2 --> SD_Pod1["2. Delete Pod mariadb-1"]
        SD_Pod1 --> SD_PVC1{"Does the PVC\ndata-mariadb-1\nget deleted too?"}
        SU_Start["Scale-Up Begins"] --> SU_Pod1["1. Create Pod mariadb-1"]
        SD_PVC1 -- "NO" --> SD_Retain1["PVC Survives (Bound) & Data Safe"]
    end

    subgraph ScaleUp["Scale-Up Cycle Back (Replicas: 1 ➔ 3)"]
        SU_Pod1 --> SU_Mount1["Kubelet Automatically Mounts the Old PVC data-mariadb-1"]
        SU_Mount1 --> SU_Ready1["Pod mariadb-1 Ready with Old Data Intact"]
        
        SU_Ready1 --> SU_Pod2["2. Create Pod mariadb-2"]
        SU_Pod2 --> SU_Mount2["Kubelet Automatically Mounts the Old PVC data-mariadb-2"]
        SU_Mount2 --> SU_Ready2["Pod mariadb-2 Ready with Old Data Intact"]
    end

    SD_Retain1 --> SU_Start

The relationship between Pod mariadb-1 and PVC data-mariadb-1 is permanent and never changes for the StatefulSet’s entire lifetime. If the worker node VM hosting mariadb-1 dies completely, the scheduler moves that Pod to another healthy worker node. The new Pod that starts still has the name mariadb-1 and is guaranteed to mount the same PVC data-mariadb-1. Our data stays consistent and safe across hardware failures.

PVC Lifecycle Inside a StatefulSet #

The PVC lifecycle behavior inside a StatefulSet is designed with very high data protection levels. Here are the mechanics in three operational scenarios:

1. Scale-Up Scenario (Adding Replicas) #

When we raise the replica count (e.g. from replicas: 1 to replicas: 3), the StatefulSet Controller processes Pod creation sequentially (OrderedReady):

  • The controller sees db-1 will be created. It checks whether the PVC data-db-1 already exists in the API Server.
  • If it doesn’t exist, it creates the data-db-1 PVC object and waits until its status becomes Bound.
  • After the physical disk is successfully created by the CSI driver and the PVC status becomes Bound, the new db-1 Pod gets scheduled and started.
  • This process repeats sequentially for Pod db-2.

2. Scale-Down Scenario (Removing Replicas) #

When we lower the replica count (e.g. from 3 to 1), the StatefulSet Controller deletes Pods starting from the highest ordinal index (db-2, then db-1, sequentially).

  • Important: When Pod db-2 is deleted, the PVC data-db-2 object and its physical cloud disk are NOT deleted along with it.
  • Design Reason: This is Kubernetes’ built-in safety rails to protect our valuable data. If we accidentally scale down a database, our data stays intact. When we scale replicas back up to 3, the new db-2 Pod automatically reconnects to the old PVC data-db-2, complete with all its historical data.

3. StatefulSet Deletion Scenario #

If we run kubectl delete statefulset db, Kubernetes deletes the StatefulSet controller object along with all its Pods. However, all PVC objects are retained.

  • To truly discard data and stop cloud disk billing, we must manually delete the PVCs:
    kubectl delete pvc -l app=mariadb -n database

Tactics for Modifying Immutable Parameters: The Cascade Orphan Method #

In production operations, we often face situations where we must change properties inside volumeClaimTemplates (e.g. changing the default disk capacity from 50Gi to 100Gi or switching the storageClassName).

  • Challenge: Kubernetes blocks direct modification of the volumeClaimTemplates property because it’s considered immutable (can’t be changed after creation). If we force kubectl apply, the API Server rejects the request with an error.

To modify these parameters without triggering database failures (zero-downtime), we can use the Cascade Orphan Deletion tactic:

Step 1: Delete the StatefulSet with the Orphan Policy #

We delete only the StatefulSet object while instructing Kubernetes to let the database Pods and their PVCs stay alive independently in the cluster:

kubectl delete statefulset mariadb-cluster --cascade=orphan -n database

After this command runs, the StatefulSet object is gone, but Pods mariadb-cluster-0, mariadb-cluster-1, and mariadb-cluster-2 keep running and serving SQL queries as usual.

Step 2: Change the Spec in the YAML Manifest Repository #

Open our StatefulSet YAML manifest file, then change the specs inside volumeClaimTemplates per the new needs (e.g. changing the StorageClass or disk size).

Step 3: Apply the New StatefulSet Manifest #

Run the apply command to recreate the StatefulSet object:

kubectl apply -f mariadb-statefulset-updated.yaml

The newly started StatefulSet Controller scans the cluster. It detects that Pods mariadb-cluster-0 through 2 are already running and have matching label selectors. The controller transparently adopts those Pods into its ownership without restarting containers. This procedure is very safe and avoids database downtime risk.

Data Migration Patterns Between PVCs (HDD to SSD Upgrade / Cross-SC) #

When we want to upgrade database storage performance (e.g. from a slow standard-hdd class to a premium premium-ssd class), we can’t just change the StorageClass name on the running PVC YAML because that property is immutable.

We must do a manual data migration process using a dedicated Job:

Step 1: Stop Database Write Traffic #

Scale the StatefulSet replicas to 0 so the database stops all I/O processes and releases its physical disk volume bindings:

kubectl scale statefulset mariadb-cluster --replicas=0 -n database

Step 2: Create a New PVC with the SSD StorageClass #

Create the new PVC object that will hold our migrated data:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-disk-mariadb-cluster-0-new
  namespace: database
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "premium-ssd-sc" # New StorageClass (SSD)
  resources:
    requests:
      storage: 50Gi                  # New capacity

Step 3: Run a Migration Job Using rsync #

We create a temporary Job object that mounts the old PVC (source) and new PVC (destination) simultaneously, then copies the entire data structure byte-for-byte using rsync to preserve file permission integrity and owner ownership:

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migration-job
  namespace: database
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrator
        image: instrumentisto/rsync:latest
        command:
        - sh
        - -c
        # Recursively copies data, preserving timestamps, permissions, and progress logs:
        - "rsync -avz --progress /mnt/source/ /mnt/dest/"
        volumeMounts:
        - name: source-vol
          mountPath: /mnt/source
          readOnly: true             # Locks the old PVC as read-only
        - name: dest-vol
          mountPath: /mnt/dest
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: data-disk-mariadb-cluster-0     # Old PVC (HDD)
      - name: dest-vol
        persistentVolumeClaim:
          claimName: data-disk-mariadb-cluster-0-new # New PVC (SSD)

Step 4: Clean Up and Rename the PVCs #

After the Job completes successfully (Completed), delete the old PVC and rename the new PVC (or recreate the StatefulSet pointing to the new PVC template).

Common StatefulSet-PVC Integration Troubleshooting #

1. Pod Stuck in Pending: “Volume Node Affinity Conflict” #

  • Symptom: Pod mariadb-cluster-1 stuck in Pending status after we restart a node. Event logs show the error: 1 node(s) had volume node affinity conflict.
  • Cause: This often happens on multi-zone cloud clusters. PVC data-disk-mariadb-cluster-1 is already bound to a physical PV permanently locked in Zone A (e.g. because the StorageClass uses Immediate mode). However, the scheduler tries to schedule Pod mariadb-cluster-1 to a worker node in Zone B because Zone A is full.
  • Solution: Change our StorageClass configuration to use volumeBindingMode: WaitForFirstConsumer. If the PV was already created in the wrong zone, we’re forced to delete the PVC and let the StatefulSet recreate it in the right zone (with the consequence of data loss if the data wasn’t backed up first).

2. PVC Stuck in Terminating Status #

  • Symptom: We run kubectl delete pvc data-disk-mariadb-cluster-0, but the PVC status stays stuck in Terminating for hours.
  • Cause: Kubernetes has a safety feature called the PVC Protection Finalizer (kubernetes.io/pvc-protection). Kubernetes firmly refuses to delete PVC metadata from etcd while the PVC is still mounted by an active Pod (even if that Pod is in Error or CrashLoopBackOff condition).
  • Solution: Find which Pod is still locking the PVC, then delete or stop that Pod. Once the Pod dies, the PVC is immediately and cleanly deleted from the etcd system.

StatefulSet & PVC Anti-Patterns #

Here are three fatal configuration mistakes that often disrupt stateful production cluster operations:

Anti-Pattern 1: Ignoring fsGroup in the SecurityContext, Triggering Permission Denied When the DB Writes Data #

Deploying a database (like PostgreSQL) where the container runs as a non-root user (ID 999), but the Kubelet mounts the volume with root access permissions (0000 or read-only).

ANTI-PATTERN: Deploying a Postgres StatefulSet Without fsGroup
// WHAT WE DO:
- Deploy a postgres StatefulSet. In the Docker image, postgres runs as user 999.
- Forget to write the `fsGroup` configuration at the Pod securityContext spec level.

// THE CONSEQUENCES IN PRODUCTION:
- Permission Denied: The Kubelet mounts the new physical disk volume with the default root owner (user ID 0).
- When the postgres process tries to write cluster initialization files to `/var/lib/postgresql/data`,
  the Linux OS rejects the transaction with the error:
  `initdb: error: directory "/var/lib/postgresql/data/pgdata" exists but is not writable`.
- Our database Pod stays stuck in CrashLoopBackOff forever.

✓ THE RIGHT SOLUTION:
- Always set the `fsGroup` property inside the StatefulSet Pod's `securityContext` block, matching
  our database's internal user ID (for Postgres, usually user ID 999):
  spec:
    securityContext:
      fsGroup: 999                   # The Kubelet automatically changes the volume mount owner to user 999 at startup

Anti-Pattern 2: Hurriedly Deleting a StatefulSet to Reset Data, Leaving Cloud Bills Behind #

Assuming that deleting the StatefulSet object also deletes the PVCs and physical disks on the cloud provider.

ANTI-PATTERN: kubectl delete statefulset mariadb-cluster to clean up data
// WHAT WE DO:
- We finish testing a large database workload (e.g. allocating 500GB disks per Pod).
- We delete the StatefulSet with `kubectl delete statefulset mariadb-cluster`.

// THE CONSEQUENCES IN PRODUCTION:
- Cloud Budget Leak: Kubernetes obediently deletes our database Pods.
  However, the 500GB PVC objects and their physical cloud volumes stay actively running in the cloud console.
- The cloud provider keeps billing us thousands of dollars per month for empty disks
  that aren't actually connected to any Pod.

✓ THE RIGHT SOLUTION:
- Always make sure to explicitly delete the PVCs after deleting the StatefulSet if we truly
  intend to discard that data:
  `kubectl delete pvc -l app=mariadb -n database`

Anti-Pattern 3: Setting podManagementPolicy: Parallel on Quorum-Based Databases #

Changing the StatefulSet Pod management policy to Parallel to speed up database cluster startup.

ANTI-PATTERN: podManagementPolicy: Parallel on a Cassandra / Consul Cluster
// WHAT WE DO:
- Deploy an Apache Cassandra database cluster with 3 replicas.
- Set `podManagementPolicy: Parallel` so all three Pods start simultaneously without queueing.

// THE CONSEQUENCES IN PRODUCTION:
- Bootstrap Failure: The three database nodes start at exactly the same time.
  Because no master node is ready to act as a *seed node* yet,
  each node rejects the initial synchronization connections from the others.
- Our database cluster experiences a split-state from startup, fails to form quorum consensus,
  and ends in a system hang.

✓ THE RIGHT SOLUTION:
- Use the default `podManagementPolicy: OrderedReady` policy for databases needing quorum bootstrap.
- Only use `Parallel` mode if our application just needs stable ordinal names and PVCs,
  but has no startup order dependency (e.g. for parallel compute workers).

Summary #

  • Stable Relationship Guarantee — The StatefulSet-PVC synergy guarantees every Pod with ordinal index $i$ always reconnects to the exact same PVC across restarts.
  • Automatic PVC Protection — Kubernetes intentionally doesn’t delete PVC objects when a StatefulSet is scaled down or deleted to protect data from human error.
  • The Cascade Orphan Trick — Use kubectl delete --cascade=orphan to modify immutable volumeClaimTemplates configuration without database downtime.
  • Migration via rsync Jobs — Use a Job with the rsync tool to move data between PVCs when upgrading StorageClasses (e.g. HDD to SSD).
  • fsGroup Unlocks Access — Always set the fsGroup parameter in the Pod securityContext so the Kubelet automatically syncs volume owner permissions to the container user.
  • Manual PVC Cleanup — Remember to always explicitly delete PVCs after deleting a StatefulSet to avoid wasting cloud storage rental bills.
  • OrderedReady for Quorum — Keep the podManagementPolicy: OrderedReady policy for clustered databases to ensure the bootstrap synchronization process succeeds.

← Previous: Databases in Kubernetes   Next: Backup & Restore →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact