Dynamic Provisioning #

When we first learn about managing storage in Kubernetes, we’re introduced to the static provisioning concept. In that model, the cluster administrator must manually create PersistentVolumes (PVs) in the cloud or local infrastructure first, then developers create PersistentVolumeClaims (PVCs) to claim those volumes.

Although this method works for small clusters, it quickly becomes a big operational bottleneck as our cluster grows. Forcing administrators to manually create PVs for every new application hampers developer agility and slows deployment processes.

This is where Dynamic Provisioning comes in as the rescue solution. With dynamic provisioning, we no longer need to manually create PVs. When developers create a PVC object, Kubernetes collaborates with an external storage driver to create the physical volume on the cloud provider or locally, then automatically creates the matching PV object in the API Server. This article thoroughly dissects how dynamic provisioning works, the CSI driver architecture, and best configuration practices for production.

The End-to-End Dynamic Provisioning Workflow #

To understand the beauty of dynamic provisioning, let’s trace its creation workflow from the first second a developer applies a PVC manifest until the volume is ready for use in a Pod.

Let’s look at a visual representation of this workflow through the following diagram:

flowchart TD
    Dev["Developer (PVC)"] -. Applies .-> API["Kubernetes API Server"]
    API -. Checks .-> SC["StorageClass Resource"]
    SC -. Directs .-> CSI["CSI External Provisioner"]
    CSI -. Calls RPC CreateVolume .-> Cloud["Cloud Provider API (e.g. AWS EBS)"]
    Cloud -. Creates physical disk .-> Disk["Physical Disk (e.g. gp3 Volume)"]
    Disk -. Reports volume-id .-> CSI
    CSI -. Creates the PV object .-> API
    API -. Performs Binding .-> Bind["PVC <=> PV (Bound)"]

Let’s break down the process above step by step:

  1. Developer Creates a PVC: The developer applies a PVC manifest listing a storageClassName name (e.g. premium-ssd) and a requested size (e.g. 50Gi).
  2. Search for a Matching PV: The Kubernetes Controller Manager checks whether a static PV in Available status meets the PVC’s criteria. If none is found, Kubernetes switches to dynamic provisioning.
  3. Evaluate the StorageClass: Kubernetes reads the StorageClass definition named premium-ssd. Inside this object, Kubernetes sees the provisioner parameter (e.g. ebs.csi.aws.com) responsible for handling disk provisioning.
  4. CSI Communication: The External Provisioner controller belonging to the CSI driver running in the cluster catches the new PVC creation. This controller sends a Remote Procedure Call (RPC) in the form of the CreateVolume function to the CSI driver plugin.
  5. Physical Storage Creation: The CSI driver calls the cloud provider’s internal API (e.g. the AWS EC2 CreateVolume API or the GCP Compute Engine API) to create a physical disk with a 50Gi size and SSD type.
  6. PV Registration: The cloud provider returns a unique volume ID (e.g. vol-0a1b2c3d...). The CSI driver then automatically creates a new PersistentVolume (PV) object in Kubernetes with that backend volume’s spec.
  7. The Binding Process: The Kubernetes Controller Manager binds the developer’s PVC object to the newly auto-created PV object. The PVC status changes to Bound.
  8. Granting Pod Access: The Pod requesting that PVC can now run on the worker node. The Kubelet on the worker node instructs the local CSI driver to attach and mount the physical disk into the container filesystem.

CSI (Container Storage Interface) Architecture #

Standardized dynamic provisioning in Kubernetes today fully rests on the Container Storage Interface (CSI). Before CSI, storage driver code was written directly inside Kubernetes’ core code (in-tree drivers). That meant if AWS or NetApp wanted to release new features for their drivers, they had to wait for the main Kubernetes release cycle, which takes months.

With CSI (out-of-tree drivers), storage vendors can develop, release, and maintain their drivers independently without touching Kubernetes core code.

A modern CSI driver consists of several controller components deployed as pods in our cluster:

Main CSI Driver Components:
  ├── [Control Plane] CSI Controller (Deployment / StatefulSet)
  │     ├── External Provisioner ──> Watches PVCs, creates/deletes physical volumes.
  │     ├── External Attacher ─────> Handles volume attach/detach to worker nodes.
  │     └── External Resizer ──────> Processes volume size expansion requests.
  └── [Data Plane] CSI Node Plugin (DaemonSet)
        └── CSI Node Agent ────────> Runs on every node, processes filesystem mounts into containers.
  1. CSI External Provisioner: This controller acts as a bridge between the Kubernetes API Server and the CSI driver. It watches new PVC objects, then calls the CSI driver’s CreateVolume function to create physical storage on the cloud or locally.
  2. CSI External Attacher: Responsible for watching VolumeAttachment objects in Kubernetes. It calls the cloud provider API to attach the physical disk to the virtual machine (worker node) where the Pod will be scheduled.
  3. CSI External Resizer: Watches for capacity size changes on PVC objects. If a developer enlarges a PVC (e.g. from 50Gi to 100Gi), this controller calls the cloud API to enlarge the physical disk without stopping the Pod.
  4. CSI Node Plugin (DaemonSet): Runs on every worker node in the cluster. This component interacts directly with the local kubelet through a UNIX domain socket. When a Pod runs on that node, this DaemonSet formats the physical disk (e.g. with ext4 or xfs) and mounts the volume into the Pod’s container directory.

StorageClass: The Dynamic Provisioning Blueprint #

The main pillar of dynamic provisioning is the StorageClass resource. A StorageClass defines the “class” of storage developers can request. We can think of a StorageClass as a storage service catalog (e.g. a fast SSD class for databases, or a cheap HDD class for backups).

Let’s study the following production StorageClass manifest with its key parameter explanations:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: production-gp3-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"

1. The provisioner Parameter #

Determines which CSI driver is responsible for creating volumes. For AWS EBS, we use ebs.csi.aws.com. For Google Cloud Persistent Disk, we use pd.csi.storage.gke.io.

2. The volumeBindingMode Parameter #

This is one of the most crucial parameters for our application stability. There are two options:

  • Immediate (Default): As soon as a PVC is created, the PV is immediately created on the cloud provider at that moment. However, on multi-zone (multi-AZ) infrastructure, this is high risk. The cloud provider might create the PV in zone ap-southeast-1a, while the Kubernetes scheduler then schedules our Pod in zone ap-southeast-1b due to CPU resource limits. Because cloud block storage volumes can’t cross zone boundaries, our Pod gets stuck forever with a VolumeAttachment failed error.
  • WaitForFirstConsumer: Kubernetes doesn’t create the PV when the PVC is created. Kubernetes waits until a Pod using that PVC gets scheduled to a specific node. After the scheduler determines which node is ideal for the Pod, the scheduler informs the CSI driver of that node’s zone. The CSI driver then creates the PV exactly in the zone where the Pod is scheduled. We must use this option for all cloud production environments.

3. The reclaimPolicy Parameter #

Determines what happens to the physical volume on the cloud provider when developers delete the PVC object in Kubernetes:

  • Delete (Default): The physical cloud volume is immediately and automatically deleted too. This is very efficient for saving costs in testing or staging environments.
  • Retain: The physical cloud volume stays preserved even after the PVC and PV objects in Kubernetes are deleted. The physical PV status changes to Released. Administrators must manually delete the physical disk after confirming the data inside is no longer needed or has been moved. This option is recommended for production databases to prevent fatal data loss from accidents.

Dynamic Provisioning in On-Premise / Bare-Metal Environments #

If we build a self-managed Kubernetes cluster on local bare-metal servers (on-premise), we don’t have automatic cloud API integrations like AWS or GCP. However, we can still enjoy dynamic provisioning features using reliable local storage solutions.

1. NFS Subdir External Provisioner #

This is the simplest solution for those of us who already have a shared storage server (NFS server) on the local network. This provisioner watches PVCs, then dynamically creates new sub-directories inside our NFS share folder to allocate to new PVs.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-dynamic-sc
provisioner: cluster.local/nfs-subdir-external-provisioner
reclaimPolicy: Delete
parameters:
  archiveOnDelete: "true" # If a PVC is deleted, rename the folder (don't delete physically right away)

2. Longhorn (Distributed Block Storage) #

Longhorn is a CNCF sandbox project developed by Rancher. It provides distributed block storage on top of our local worker nodes. Longhorn replicates our block storage data to several different worker nodes to guarantee high availability.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ssd
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "3" # Data duplicated to 3 different nodes
  staleReplicaTimeout: "2880"

Anti-Patterns vs Solutions in Dynamic Provisioning #

Let’s study some common architectural mistakes (anti-patterns) when implementing dynamic provisioning, along with their code comparisons.

Anti-Pattern 1: Using Immediate Binding Mode in a Multi-Zone Cloud #

We deploy an SSD StorageClass on multi-zone AWS EKS (Singapore) with the default Immediate binding mode. This triggers a mismatch between where the physical disk is created and where the Pod gets scheduled.

Wrong Manifest Code (Immediate Binding Mode) #

# DON'T USE THIS ON MULTI-ZONE CLUSTERS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: unsafe-immediate-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: Immediate # Volume created before the Pod is scheduled!
parameters:
  type: gp3

Solution Code (Using WaitForFirstConsumer) #

# SOLUTION: Delaying volume creation until the Pod's node is determined
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: safe-wait-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer # Volume created exactly in the Pod's zone
parameters:
  type: gp3

Anti-Pattern 2: Removing Volume Expansion Support (allowVolumeExpansion) #

We forget to enable the volume expansion feature on the StorageClass. When our database runs out of disk space in the middle of the night, we’re forced to shut down the database, create a new PVC, manually move old data to the new volume, triggering long operational downtime.

Wrong Manifest Code (Without the Expansion Property) #

# DON'T USE THIS: Makes disk capacity upgrades difficult
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: no-expansion-sc
provisioner: pd.csi.storage.gke.io
# allowVolumeExpansion is not defined (default: false)

Solution Code (StorageClass with Expansion Enabled) #

# SOLUTION: Enable dynamic volume expansion
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: flexible-expansion-sc
provisioner: pd.csi.storage.gke.io
allowVolumeExpansion: true # Allows live volume resizing!

If developers want to enlarge a PVC disk from 50Gi to 100Gi, they just edit the running PVC manifest without deleting it:

# How to resize a PVC (just apply the size change directly)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-db-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: flexible-expansion-sc
  resources:
    requests:
      storage: 100Gi # Change from 50Gi to 100Gi, then run kubectl apply

Dynamic Provisioning Troubleshooting Guide #

When dynamic provisioning fails to work, developers usually find their PVCs stuck in Pending status forever. Here’s a systematic guide to diagnosing the problem:

1. Check the PVC Event Descriptions #

Our analysis starting point is checking the event logs from the PVC object itself:

kubectl describe pvc <pvc-name> -n <namespace>

Let’s study some common error messages with their explanations:

  • "waiting for a volume to be created, either by external provisioner or manually created by system administrator":
    • Meaning: The Kubernetes API Server already sent the volume creation request to the CSI driver, but the CSI driver isn’t responding or isn’t active.
    • Solution: Check whether the CSI controller pod in the system namespace (e.g. kube-system) is running or crashed:
      kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
  • "failed to provision volume: RPC error: code = InvalidParameter desc = KMS key not found":
    • Meaning: The KMS key encryption parameter written in the StorageClass is wrong or can’t be found in our cloud account.
    • Solution: Check the correctness of the KMS key ARN string in the StorageClass parameters.
  • "AccessDenied: User: arn:aws:sts::... is not authorized to perform: ec2:CreateVolume":
    • Meaning: The CSI driver pod doesn’t have sufficient IAM role/Service Account permissions to call the cloud provider API.
    • Solution: Review our IAM OIDC provider configuration and make sure the ebs-csi-controller-sa Service Account has the IAM Role ARN annotation with the proper AWS EBS permissions policy.

Summary #

  • Dynamic provisioning eliminates manual bottlenecks: PVs are automatically created by Kubernetes thanks to the collaboration between StorageClass and CSI drivers when detecting new PVC objects.
  • WaitForFirstConsumer is a multi-zone obligation: Avoid instant binding (Immediate) on multi-zone cloud providers to ensure physical volumes are created in the same zone as our Pod’s node placement.
  • Understand reclaimPolicy for data protection: Set reclaimPolicy: Retain on our important database storage so physical cloud volumes aren’t automatically deleted when PVC objects are accidentally deleted.
  • Enable allowVolumeExpansion: Always enable dynamic volume expansion on production StorageClasses so operations teams can enlarge disk capacity live without triggering database downtime.
  • Diagnose stuck PVCs through event logs: Always use kubectl describe pvc to identify IAM authorization blocks, StorageClass naming errors, or internal CSI driver communication failures.

← Previous: Backup & Restore   Next: Storage Anti-Patterns →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact