PersistentVolume #

In a Kubernetes cluster, we strictly distinguish between the Cluster Administrator role (Platform Engineer) and the Application Developer role. Administrators are responsible for managing physical infrastructure — including servers, networking, and storage capacity. Meanwhile, developers focus on code functionality and their workload needs. To support this separation of responsibilities on the persistent data storage aspect, Kubernetes provides the PersistentVolume (PV) object.

A PersistentVolume is the physical representation of storage media prepared at the cluster level. Just as Nodes represent the cluster’s global compute capacity (CPU/RAM), PVs represent the global storage capacity not tied to any namespace (non-namespaced resource). This article covers the PV manifest anatomy in depth, performance-determining parameters, the PV status lifecycle, the static provisioning mechanism, and how to recover data locked in Released status.

PersistentVolume’s Position in the Storage Ecosystem #

To understand how PersistentVolumes work, we must see how PVs interact with PersistentVolumeClaims (PVCs) and StorageClasses. A PVC acts as a request (“claim”) from developers, while a PV acts as the capacity provider ready to be bound to that PVC.

Here’s a diagram of the PersistentVolume status lifecycle flow from creation to cleanup:

flowchart TD
    Created["PV Created (Available)"] --> BindAction["New PVC Requests a Bind"]
    BindAction --> Bound["PV Bound to a PVC (Bound)"]
    
    Bound --> PVC_Delete["User Deletes the PVC Object"]
    PVC_Delete --> CheckPolicy{"What Is the\nReclaim Policy?"}
    
    CheckPolicy -- "Retain" --> Released["Released (Physical Data Safe, PV Locked)"]
    CheckPolicy -- "Delete" --> Deleted["Cloud Disk Deleted & PV Deleted"]
    CheckPolicy -- "Recycle" --> Clean["rm -rf on the Directory"]
    
    Clean --> Created
    
    subgraph AdminAction["Manual Admin Intervention"]
        Released --> AdminDecide{"What Action?"}
        AdminDecide -- "Remove claimRef" --> Created
        AdminDecide -- "Delete PV & Data" --> Finish["Deletion Complete"]
    end

PersistentVolume Manifest Anatomy #

To understand the high-level configuration parameters PVs provide, let’s look at a manifest showing a manually configured NFS-server-based PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-production-postgres
  labels:
    storage-tier: premium
    encrypted: "true"
spec:
  capacity:
    storage: 100Gi                   # Physical disk capacity
  volumeMode: Filesystem             # Filesystem (Default) | Block
  accessModes:
    - ReadWriteOnce                  # RWO | ROX | RWX | RWOP
  persistentVolumeReclaimPolicy: Retain # Retain | Delete | Recycle
  storageClassName: premium-manual   # Matched with the PVC's storageClassName
  nfs:                               # Storage backend type (NFS)
    path: /var/exports/postgres-data
    server: nfs-server.production.internal

Key Parameter Explanations: #

1. capacity.storage #

Defines the physical storage capacity allocated to this volume (e.g. 100Gi or 100 Gibibytes). Kubernetes uses standard IEC binary units for storage sizes.

2. volumeMode #

Determines how the volume is presented into the container:

  • Filesystem (Default): The volume is formatted with a filesystem (like ext4 or xfs) before being mounted into the container. This is the standard option for most applications.
  • Block: The volume is presented as a raw block device without any filesystem. This option is very useful for high-performance database applications (like Oracle, Apache Cassandra, or advanced PostgreSQL) that want to manage block I/O themselves without Linux filesystem overhead.

3. persistentVolumeReclaimPolicy #

The policy determining what Kubernetes should do to the physical storage media after the PVC object binding it is deleted:

  • Retain (Recommended for Production): When the PVC is deleted, the PV changes to Released status. The physical data in storage stays safe and isn’t deleted. The cluster administrator must do manual verification before cleaning that data.
  • Delete: When the PVC is deleted, the PV object and the physical storage media on the cloud provider (like AWS EBS volumes) are automatically deleted. This is the default for volumes automatically created by StorageClass (dynamic provisioning).
  • Recycle (Deprecated): Runs a basic cleanup operation (rm -rf /volume/*) on the NFS volume to clear old data, then returns the PV status to Available so it can be reclaimed. This feature has been fully replaced by the safer dynamic provisioning mechanism.

Understanding PV Status Lifecycle Phases #

A PersistentVolume object moves dynamically through statuses in the cluster’s etcd database following user interactions:

1. Available #

The PV has been successfully created by the Administrator or Storage Controller and is ready for use. No PVC object has bound it yet.

2. Bound #

The PV has been successfully matched and bound exclusively to a PVC object. While in this status, no other PVC may claim this PV. The PV stays Bound until that PVC object is deleted by the user.

3. Released #

The user has deleted the companion PVC object. Because the PV reclaim policy is Retain, Kubernetes breaks the binding relationship but doesn’t delete the physical data. A Released PV stays administratively locked.

  • Important: A Released PV can’t be directly reclaimed by a new PVC, because the PV metadata still stores a reference to the old PVC (spec.claimRef).

4. Failed #

A failure occurred when Kubernetes tried to do automatic deletion (with the Delete policy) or cleanup (with the Recycle policy). This status requires manual intervention from the Platform Engineering team to diagnose the problem in the CSI storage driver.

Tactical Guide: Reusing PVs in Released Status #

In production environments, we often face situations where developers accidentally delete a PVC. Because the reclaim policy is set to Retain, our data is still safe on the physical disk, but the PV status stays stuck in Released. How do we return that PV to reusable status for a new PVC without losing data?

Here are the steps to recover a Released PV back to Available:

Step 1: Check the PV Status #

We see the PV stuck in Released status with a leftover reference to the old PVC:

kubectl get pv pv-production-postgres

Output:

NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
pv-production-postgres   100Gi      RWO            Retain           Released   production/postgres-pvc-old

Step 2: Manually Remove the claimRef Block #

The key locking that PV is the spec.claimRef property in the PV manifest. We can remove this property by running the edit command:

kubectl edit pv pv-production-postgres

In the text editor that appears, find the claimRef block under spec and remove the entire block:

# BEFORE:
spec:
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: postgres-pvc-old
    namespace: production
    resourceVersion: "123456"
    uid: "abc-123-def"

# AFTER EDITING (Remove the entire claimRef block above):
spec:
  # The claimRef property has been removed

Step 3: Verify the PV Status #

After saving the changes, the kube-controller-manager detects the missing claimRef and immediately changes the PV status back to Available:

kubectl get pv pv-production-postgres

Output:

NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM
pv-production-postgres   100Gi      RWO            Retain           Available

Now application developers can create a new PVC pointing to this PV’s storageClassName or labels to safely reconnect the application to the old data.

Static Provisioning vs Dynamic Provisioning #

There are two ways to provide PersistentVolume storage in Kubernetes:

1. Static Provisioning (Manual Provisioning) #

The cluster administrator must predict future storage needs. They create the physical disk in the cloud console, note the volume ID, then write PV YAML manifests manually one by one.

  • Problem: Creates an operational bottleneck. If developers need a new database in the middle of the night, they must wait for the Administrator to come to work and create the PV first. If no matching PV exists, the PVC stays stuck in Pending status forever.

2. Dynamic Provisioning (Automatic Provisioning) #

The modern mechanism where the Kubernetes cluster creates the physical disk and PV object automatically in real-time as soon as a new PVC is declared by developers. This is made possible by the StorageClass object.

  • Advantage: Full self-service efficiency. Developers just create a PVC, and the StorageClass orders the CSI driver to call the cloud API to provision physical storage automatically within seconds.

Node Affinity on Local PersistentVolumes #

In some special scenarios (like extreme-performance database clusters), we want to use NVMe SSD physical disks physically attached directly inside the worker node machine (local storage), not cloud disks accessed over a virtual network.

Because the data is stored on a specific machine’s physical disk, we must tell Kubernetes not to move Pods using this storage to another machine. We do this by writing a Node Affinity configuration in the local PV manifest:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-ssd-pv-01
spec:
  capacity:
    storage: 500Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-fast-ssd
  local:
    path: /mnt/disks/nvme-ssd-1       # Mount path on the worker node host
  nodeAffinity:                       # Must be defined!
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-3             # This local disk ONLY exists on worker-node-3

[!IMPORTANT] When using Local PVs with Node Affinity, we must set the volumeBindingMode: WaitForFirstConsumer property on the relevant StorageClass. If we use the default Immediate mode, Kubernetes immediately binds the PVC to the local PV before the Pod is scheduled, which often ends in failure if that Pod turns out to be unschedulable on worker-node-3 due to node CPU or memory limits.

PersistentVolume Anti-Patterns & Their Solutions #

Here are three fatal mistakes that often cause data chaos at the administrative level:

Anti-Pattern 1: Using the Delete Reclaim Policy for Manually Provisioned Production Databases #

Setting persistentVolumeReclaimPolicy: Delete on an important database’s manual PV manifest.

ANTI-PATTERN: persistentVolumeReclaimPolicy: Delete for a Manual Postgres DB
// WHAT WE DO:
- Create a manual PV for the database server with the Delete policy.
- A developer accidentally runs a namespace deletion command (kubectl delete namespace production)
  or deletes the PVC for testing.

// THE CONSEQUENCES IN PRODUCTION:
- Instant Data Loss: Kubernetes immediately detects the PVC object deletion.
- Because the policy is set to `Delete`, the controller sends instructions to the cloud provider
  to delete the physical volume (EBS/GCP PD) from the cloud console.
- All production transaction data is instantly destroyed and cannot be recovered from the cluster.

✓ THE RIGHT SOLUTION:
- Always use `persistentVolumeReclaimPolicy: Retain` on all manually created PV manifests.
- If volumes are dynamically created by a StorageClass, enable deletion protection features
  at the database level or on the cloud provider side.

Anti-Pattern 2: Putting Stateful Data on hostPath PVs Without Node Affinity #

Creating manual PVs using the hostPath driver type without a nodeAffinity rule locking the worker node location.

ANTI-PATTERN: A hostPath PersistentVolume Without nodeAffinity
// WHAT WE DO:
- Create a PV named `data-pv` with the `hostPath` driver pointing to `/mnt/data` on the host.
- Don't write a `nodeAffinity` block in the PV manifest.

// THE CONSEQUENCES IN PRODUCTION:
- Data Inconsistency: The Scheduler assumes this PV can be used by Pods on any node.
- When the Pod is scheduled on `worker-node-1`, it reads data in `worker-node-1`'s `/mnt/data`.
- The next day, the Pod restarts and gets scheduled on `worker-node-2`. It mounts `/mnt/data`
  on `worker-node-2`, whose contents might be empty or hold garbage files from another application,
  causing the app to crash or misread state data.

✓ THE RIGHT SOLUTION:
- Don't use `hostPath` for general stateful PVs.
- If forced to use worker node local storage, use the **`local`** volume type
  and must include a `nodeAffinity` declaration locking the specific worker node hostname.

Anti-Pattern 3: Letting Released PVs Accumulate in the Cluster Without Cleanup #

Ignoring cluster capacity monitoring so many Released PVs get abandoned in the etcd database.

ANTI-PATTERN: Letting Released PVs Rot Forever
// WHAT WE DO:
- Developer teams frequently deploy and delete test applications (CI/CD pipelines).
- Because the PV policy uses `Retain`, every deletion leaves one `Released` PV object behind.
- We have no automatic or manual cleanup procedure.

// THE CONSEQUENCES IN PRODUCTION:
- Cloud Budget Waste: Physical cloud disks in the backend stay actively running and keep charging our monthly cloud bill.
- PVC Starvation: New developers try to deploy applications, but their PVCs stay stuck in `Pending` status
  because all remaining PVs are `Released` (locked); no `Available` PV is ready to claim.

✓ THE RIGHT SOLUTION:
- Periodically audit cluster PV objects with: `kubectl get pv | grep Released`.
- Create an out-of-cluster cleanup cron job detecting `Released` PVs older than 3 days,
  then cleanly send deletion instructions to both the cloud API and the Kubernetes API Server.

Summary #

  • A Global Non-Namespaced Abstraction — A PersistentVolume is the cluster’s physical storage representation at the administrative level, global and not tied to namespaces.
  • Data Protection via Retain — Always use persistentVolumeReclaimPolicy: Retain for production data so physical disk data isn’t deleted along with the PVC object.
  • volumeMode Block for DBs — Use the volumeMode: Block setting for high-performance intensive databases to bypass Linux filesystem formatting overhead.
  • Released Status Resolution — Return Released PVs to Available by removing the spec.claimRef reference block in the PV manifest.
  • Local PVs Require Node Affinity — When using local NVMe storage, include Node Affinity on the PV to guarantee Pods always schedule on the right physical machine.
  • Dynamic Provisioning for Scale — Use StorageClass (dynamic provisioning) to avoid the administrative bottleneck of manual PV creation (static provisioning).
  • Avoid Wild hostPath — Don’t deploy hostPath PVs without nodeAffinity to avoid data read collisions across different worker nodes.

← Previous: Storage Problems in Distributed Systems   Next: PersistentVolumeClaim →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact