PersistentVolumeClaim #

In the Kubernetes ecosystem, application developers ideally shouldn’t be bothered by storage infrastructure technical details — like the SAN hardware type used, NFS server IP addresses, or cloud provider API credentials for creating new disks. Developers only need to declare their application’s storage needs logically: what capacity is needed, how the data will be accessed, and what storage performance type is desired.

To fulfill those needs, Kubernetes provides the PersistentVolumeClaim (PVC) object. A PVC acts as a request ticket or “claim” over storage space submitted by users at the namespace level (namespaced resource). Kubernetes then takes full responsibility for finding a matching physical PersistentVolume (PV), or automatically creating one via a StorageClass. This article dissects the PVC manifest anatomy in depth, the binding matching algorithm, implementation in Pods and StatefulSets, PVC troubleshooting guides for stuck volumes, and the capacity scaling mechanism (volume expansion).

The Binding Flow: How PVCs Find Physical PVs #

When we create a new PVC object, it starts in Pending status. Kubernetes’ internal controller (specifically the volume binding controller) periodically watches new PVCs and tries to pair them with physical PersistentVolume (PV) objects in Available status.

Here’s the logical decision flow Kubernetes follows to connect a PVC to a PV:

flowchart TD
    PVC_Create["PVC Created (Pending)"] --> SC_Check{"Is\nstorageClassName\ndefined?"}
    
    SC_Check -- "No / Empty" --> StaticSearch["Search for a Matching Manual Available PV"]
    StaticSearch --> MatchCheck{"Was a matching\nPV found?"}
    
    MatchCheck -- "Yes" --> StaticBind["Perform Binding (Status: Bound)"]
    MatchCheck -- "No" --> StuckPending["Stuck in Pending (Waiting for an Admin-Created PV)"]
    
    SC_Check -- "Yes (Class Name)" --> DynamicCheck{"Does the StorageClass\nsupport dynamic\nprovisioning?"}
    
    DynamicCheck -- "Yes" --> ProvisionStart["Send an API Request to the CSI Driver"]
    ProvisionStart --> CloudDisk["Cloud Provider Creates the Physical Disk"]
    CloudDisk --> PV_Auto["Kubernetes Creates the PV Object Automatically"]
    PV_Auto --> DynamicBind["Perform Automatic Binding (Status: Bound)"]
    
    DynamicCheck -- "No" --> StaticSearch

The Strict Matching Criteria: #

A PV can only be declared a match for binding to a PVC if it meets all of the following criteria:

  1. StorageClass Match: The PVC’s storageClassName value must match the storageClassName written on the PV.
  2. Sufficient Capacity: The PV’s physical storage capacity must be greater than or equal to (>=) the request value from the PVC.
  3. Compatible Access: The accessModes supported by the PV must include the access mode requested by the PVC (e.g. if the PVC requests ReadWriteMany, the PV must also support ReadWriteMany).
  4. Ready Status: The PV status must be in the Available phase (not yet bound to another PVC).

[!IMPORTANT] The Kubernetes binding system uses a first-fit matching algorithm, not best-fit matching. If we create a PVC requesting 10Gi capacity, and the cluster has two Available PVs: pv-a sized 100Gi and pv-b sized 10Gi, Kubernetes might bind our PVC to the 100Gi pv-a. This wastes the remaining 90GiB capacity that ultimately can’t be used by anyone. To avoid this, we can lock the binding to a specific PV using the spec.volumeName property on the PVC.

PersistentVolumeClaim Manifest Anatomy #

Here’s a production-grade PVC manifest requesting a premium storage class volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data-pvc
  namespace: production              # PVCs are namespaced
spec:
  accessModes:
    - ReadWriteOnce                  # Desired access mode
  volumeMode: Filesystem             # Filesystem | Block
  resources:
    requests:
      storage: 50Gi                  # Minimum capacity requested
  storageClassName: premium-ssd-sc   # Points to the dynamic provisioning StorageClass
  selector:                          # Optional: Select static PVs by label
    matchLabels:
      tier: fast

Key Parameter Explanations: #

  • namespace: The PVC is locked inside a specific namespace. Only Pods running in the same namespace as the PVC are allowed to mount that volume.
  • resources.requests.storage: The minimum capacity limit requested by the application. If the StorageClass triggers dynamic cloud disk creation, the physical disk created will match this value’s size.
  • selector: Used when we want to do static PV matching (static provisioning) based on marker labels on the PV, instead of using dynamic provisioning from the StorageClass.

Using PVCs in the Pod Spec #

Once the PVC status changes to Bound, we can directly use it as a volume in the Pod spec manifest. We do this by adding the spec.volumes[*].persistentVolumeClaim block pointing to our PVC name:

apiVersion: v1
kind: Pod
metadata:
  name: postgres-app
  namespace: production
spec:
  containers:
  - name: postgres
    image: postgres:15
    ports:
    - containerPort: 5432
    volumeMounts:
    - name: pg-storage               # Points to the volume name below
      mountPath: /var/lib/postgresql/data
  volumes:
  - name: pg-storage
    persistentVolumeClaim:
      claimName: pg-data-pvc         # The PVC name we created earlier
      readOnly: false

Independent Lifecycles: #

One thing we must understand: PVCs and Pods have absolutely separate lifecycles.

  • If the postgres-app Pod above is deleted, the pg-data-pvc object and all transaction data on the physical disk stay safe.
  • When we create a new Pod pointing to the same pg-data-pvc, the new Pod immediately reads the inherited old data. This is the resilience foundation of stateful application data.

PVC Automation Pattern on StatefulSets: volumeClaimTemplates #

If we use a Deployment object to deploy an application with many replicas (e.g. 3 replicas), and that Deployment mounts a regular PVC, all three Pod replicas try to mount the exact same PVC. If the PVC is ReadWriteOnce type, the second and third Pods get stuck and fail to start due to access restrictions.

To fulfill distributed database cluster needs requiring a unique disk for each Pod replica, Kubernetes provides the StatefulSet feature with the volumeClaimTemplates property:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-cluster
  namespace: database
spec:
  serviceName: "mariadb-headless"
  replicas: 3
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb:10.11
        volumeMounts:
        - name: data-disk            # Must match the name in the template below
          mountPath: /var/lib/mysql
  volumeClaimTemplates:              # Automatically creates a unique PVC per Pod!
  - metadata:
      name: data-disk
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "gcp-ssd-sc"
      resources:
        requests:
          storage: 20Gi

StatefulSet PVC Automatic Naming Mechanism: #

When the StatefulSet above starts, the Kubernetes controller automatically creates 3 independent PVCs in the API Server with the name format: $$\text{PVC Name} = \text{{template-name}}-\text{{statefulset-name}}-\text{{ordinal-index}}$$

Automatically created PVCs:
  - data-disk-mariadb-cluster-0 (exclusively mounted by Pod mariadb-cluster-0)
  - data-disk-mariadb-cluster-1 (exclusively mounted by Pod mariadb-cluster-1)
  - data-disk-mariadb-cluster-2 (exclusively mounted by Pod mariadb-cluster-2)

If Pod mariadb-cluster-1 crashes and moves nodes, it’s guaranteed to always be re-paired with PVC data-disk-mariadb-cluster-1. Data will never get mixed up between database cluster members.

Troubleshooting Guide: Fixing PVCs Stuck in Pending Status #

The most common operational problem developer teams complain about is PVCs staying stuck in Pending status for a long time. We must investigate systematically to find the root cause.

1. Run the Initial Diagnostic Command #

Use the describe command to read the event logs recorded by the API Server on the PVC object:

kubectl describe pvc <pvc-name> -n <namespace>

Direct our attention to the very bottom of the output, specifically the Events column. Here we’ll find the diagnostic message explaining the binding failure cause.

2. Read the Error Messages and Their Solutions #

Here are common PVC event error message patterns with their solutions:

Case A: “no persistent volumes available for this claim and no storage class is set” #

  • Meaning: We didn’t define storageClassName in the PVC manifest, while the cluster has no Available PersistentVolume (PV) with enough size, and the cluster has no active default StorageClass.
  • Solution: Create a physical PV manually, or contact the Platform Engineer to register one of the StorageClasses as the cluster default using the annotation: storageclass.kubernetes.io/is-default-class: "true"

Case B: “waiting for a volume to be created, either by external provisioner… or manually created by system administrator” #

  • Meaning: The PVC successfully points to a valid StorageClass, but the automatic provisioning process is stuck. This is usually because the CSI driver pod (like ebs.csi.aws.com) in the kube-system namespace crashed or lacks the IAM access permissions (cloud credentials) to create new disks.
  • Solution: Check the CSI Driver pod health status: kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver Check that driver pod’s logs for cloud API authorization failures.

Case C: “volume node affinity conflict” #

  • Meaning: This happens on Local PVs or multi-zone storage volumes. The kube-scheduler detects that the physically available PV is in Zone A, but the Pod submitting the PVC can only be scheduled in Zone B due to taints or CPU scarcity in Zone A.
  • Solution: Loosen the node affinity restrictions on the Pod, or make sure we create the new storage volume in the same zone as the healthy worker node location.

Volume Expansion: Online Storage Capacity Scaling #

Over time, our database data will definitely grow. Kubernetes provides the Volume Expansion feature, letting us enlarge storage size directly (online resizing) without destroying Pods or taking down services.

1. Verify Support on the StorageClass #

Before making changes, make sure the StorageClass managing our PVC allows the expansion feature with the allowVolumeExpansion: true parameter:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium-ssd-sc
provisioner: pd.csi.storage.gke.io   # Example GCE PD CSI
allowVolumeExpansion: true            # Must be true!

2. Change the PVC Size Declaratively #

We just edit the PVC manifest or run a patch command to raise the request capacity:

kubectl patch pvc pg-data-pvc -n production \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

3. Monitor the Filesystem Resizing Process #

After the cloud provider API finishes enlarging the physical disk capacity, the Kubelet on the worker node detects the change and runs the container’s internal filesystem expansion operation. We can monitor this status through events:

kubectl describe pvc pg-data-pvc -n production

Look for the Conditions section in the output. If the resize succeeds, we’ll see the message: VolumeExpansionSuccessful: Mount volume expanded successfully on node.

[!WARNING] Volume scaling is one-way only: GROWING. We can never shrink a PVC size (e.g. lowering from 100Gi to 50Gi) because Linux filesystems (like ext4/xfs) don’t support online size reduction and it risks destroying the entire data integrity on the disk.

PVC Management Anti-Patterns & Their Solutions #

Here are three PVC configuration mistakes that most often cause operational failures in production:

Anti-Pattern 1: Ignoring storageClassName (Using a Hidden Default) #

Leaving the storageClassName property empty on a PVC without realizing the default binding consequences.

ANTI-PATTERN: Writing storageClassName Without Verifying the Default Class
// WHAT WE DO:
- Write a PostgreSQL database PVC manifest without including the `storageClassName` field.

// THE CONSEQUENCES IN PRODUCTION:
- Unexpected Binding: Kubernetes automatically binds our PVC to the cluster's default StorageClass.
  On managed clouds (like GKE/EKS), the default StorageClass usually uses a slow standard disk type (HDD).
- I/O Degradation: Our database runs slow from the first day of deployment because it uses HDD storage,
  slowing API response times when accessed by many users.

✓ THE RIGHT SOLUTION:
- Always explicitly declare the `storageClassName` property on every PVC manifest:
  storageClassName: "premium-ssd-sc"
- If we intentionally want to use a manually created static PV, explicitly set the property to an empty string:
  storageClassName: ""

Anti-Pattern 2: Sharing a ReadWriteOnce (RWO) PVC Across Nodes on Stateless Deployments #

Deploying a 3-replica web API Deployment using an RWO PVC for a shared log cache directory.

ANTI-PATTERN: 3-Replica Web API Using a Shared RWO PVC
// WHAT WE DO:
- Deploy a web API with 3 Pod replicas.
- Configure all three Pods to mount one RWO PVC named `web-log-pvc`.

// THE CONSEQUENCES IN PRODUCTION:
- Startup Failure: The Scheduler schedules `pod-0` on `node-1`, `pod-1` on `node-2`, and `pod-2` on `node-3`.
- `pod-0` on `node-1` runs normally because it manages to lock the cloud disk volume.
- `pod-1` and `pod-2` on the other worker nodes stay stuck in `ContainerCreating` status forever with the error:
  `Multi-Attach error for volume: volume-xyz can only be attached to single node`.

✓ THE RIGHT SOLUTION:
- If web API logs need to be collected, dump logs to stdout/stderr, don't store them on a shared disk.
- If forced to need a shared folder across worker nodes, use a PVC with `ReadWriteMany` (RWX) access mode
  and make sure to use network storage backends like AWS EFS or NFS.

Anti-Pattern 3: Trying to Shrink PVC Capacity via GitOps/YAML Patches #

Trying to reduce the PVC storage size (e.g. from 100Gi to 50Gi) through a GitOps repository update.

ANTI-PATTERN: Changing storage: "100Gi" to storage: "50Gi" in the PVC YAML
// WHAT WE DO:
- Realize the 100Gi allocation is too big and want to save cloud costs by changing the manifest to 50Gi.

// THE CONSEQUENCES IN PRODUCTION:
- API Server Rejection: The Kubernetes API Server instantly rejects that configuration submission.
- Stuck GitOps Pipeline: The CI/CD system (like ArgoCD) stays stuck in `OutOfSync` or `Error` status
  because Kubernetes refuses to apply the smaller storage property change, halting the entire automated deployment flow.

✓ THE RIGHT SOLUTION:
- Accept the reality that physical volumes can't be shrunk.
- If capacity needs to be reduced, create a new 50Gi PVC, deploy a temporary data migration Pod
  to move data from the 100Gi PVC to the 50Gi PVC (e.g. using rsync),
  then delete the old 100Gi PVC once it's safe.

Summary #

  • The Namespace Claim Ticket — A PersistentVolumeClaim (PVC) is the abstract representation of a storage request from developers, locked inside a specific namespace.
  • First-Fit Matching — Kubernetes uses a first-fit algorithm to match PVCs with Available PVs; use volumeName if you want to lock the binding to a specific PV.
  • Separate Lifecycles — Pods using a PVC can be freely deleted and recreated without affecting the PVC object’s integrity or the physical data on the disk.
  • StatefulSet Automation — Use the volumeClaimTemplates block on StatefulSets to automatically generate unique PVCs for each Pod replica.
  • describe PVC for Diagnosis — Always run kubectl describe pvc and check the Events section to identify why a PVC is stuck in Pending status.
  • allowVolumeExpansion — Enable the allowVolumeExpansion: true parameter on the StorageClass so PVCs can be enlarged online without downtime.
  • Volumes Can Only Grow — Never try to lower the PVC storage capacity in YAML manifests to avoid rejection errors from the API Server.
  • Default Class Alert — Always explicitly specify storageClassName on PVCs to avoid misallocation to a low-performance default StorageClass.

← Previous: PersistentVolume   Next: StorageClass →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact