Storage Performance #

When we design application architecture in Kubernetes, our attention often focuses entirely on optimizing CPU and RAM memory allocation. We spend lots of time tuning Vertical Pod Autoscalers or precisely determining container resource limits. However, there’s one important component that often escapes attention until major problems hit the production environment: storage performance.

Storage performance is a key determinant of our stateful applications’ stability. Slow databases, failed queue log writes, ballooning API latency, and cascading transaction timeouts are often not caused by a lack of CPU or memory, but by I/O limitations in the storage subsystem.

In this article, we’ll thoroughly dissect the crucial storage performance metrics, I/O profiles for various workloads, high-performance local storage integration, the influence of cluster networking on performance, and how to run benchmark tests independently inside our Kubernetes cluster.

The Three Main Storage Performance Metrics #

To understand storage characteristics, we must recognize the three fundamental metrics determining disk performance. These three metrics don’t stand alone; they influence each other and shape a storage volume’s performance profile.

flowchart TD
    Metrics["STORAGE PERFORMANCE METRICS"]
    Metrics --> IOPS["IOPS (Number of R/W operations per second)"]
    Metrics --> Throughput["Throughput (Volume of data transferred per second)"]
    Metrics --> Latency["Latency (Response time of one read/write operation)"]

1. IOPS (Input/Output Operations Per Second) #

IOPS measures the number of read or write operations a disk can complete in one second. This metric is critical for applications frequently performing small random transactions (random I/O). Transactional databases (OLTP) like PostgreSQL, MySQL, and search systems like Elasticsearch heavily depend on high IOPS to process thousands of small queries per second.

2. Throughput (Bandwidth) #

Throughput measures the total data volume that can be read from or written to the storage media per unit time (usually in MegaBytes per second or MB/s). Throughput is very important for workloads reading or writing large files sequentially (sequential I/O). Examples of throughput-hungry workloads are log streaming systems (fluentd/Loki), large-scale batch data processing, Kafka message brokers, and database backup restores.

3. Latency #

Latency is the time the storage subsystem needs to complete a single I/O request (measured in milliseconds or ms, or microseconds or µs). For transactional databases, the write latency of fsync operations (writing data from memory cache to the physical disk platter for data durability) is the most crucial metric. If latency is high, every database transaction stalls, triggering connection buildup and system failures.

I/O Profiles by Workload #

Every stateful application has unique I/O characteristics. We can’t use one StorageClass type for all workload kinds. Here’s an I/O profile guide for various common applications we run in Kubernetes:

1. Relational Databases (PostgreSQL & MySQL) #

• Characteristics: High random I/O, small block sizes (usually 8KB to 16KB per operation), sensitive to fsync latency.
• Main Needs: Very low latency (sub-millisecond) and stable high IOPS.
• StorageClass Recommendation: Premium SSDs with defined IOPS (e.g. AWS gp3 type with increased IOPS of at least 3000-6000, or the io2 type for very busy financial databases).

2. Message Brokers (Apache Kafka) #

• Characteristics: Very high sequential write I/O, append-only distributed log file write operations.
• Main Needs: Large sequential throughput and generous disk capacity. Random IOPS isn’t that critical because Kafka writes data to memory page cache first before flushing it to disk sequentially.
• StorageClass Recommendation: Standard SSD types (gp3 with high throughput) or throughput-optimized HDDs (st1 AWS) for Kafka clusters with long data retention.

3. Log Aggregators (Elasticsearch / OpenSearch) #

• Characteristics: A combination of random I/O (during index searches) and high sequential I/O (when continuously indexing new logs and merging shards).
• Main Needs: Consistent write throughput and medium-high IOPS.
• StorageClass Recommendation: SSDs with enhanced throughput parameters (e.g. gp3 with throughput set to 250-500 MB/s).

Local Persistent Volumes for Maximum Performance #

For ultra-fast transactional database workloads needing the lowest latency response times (under 1 millisecond), network-attached storage (like AWS EBS or Google Cloud Persistent Disk) sometimes isn’t sufficient anymore due to network overhead latency.

Our best solution for this case is using Local Persistent Volumes (LPVs), leveraging physical NVMe disks attached directly inside our worker node’s physical server.

Let’s look at the I/O access path difference between Local Storage and Network Storage through the following diagram:

flowchart TD
    subgraph LocalStoragePath["Local Storage Path (Ultra-Low Latency)"]
        PodLocal["Application Pod"] --> NodeLocal["Worker Node (OS Kernel)"]
        NodeLocal --> PhysicalNVMe["Local Physical NVMe Disk"]
    end
    
    subgraph NetworkStoragePath["Network Storage Path (Many Network Hops)"]
        PodNet["Application Pod"] --> NodeNet["Worker Node (OS Kernel)"]
        NodeNet --> NetworkSwitch["Cluster Network Switch"]
        NetworkSwitch --> StorageController["Cloud Storage Controller"]
        StorageController --> NetworkDisk["Physical Cloud Disk (e.g. EBS)"]
    end

Implementing Local Persistent Volumes #

Because local volumes are tightly bound to one specific physical node, Kubernetes can’t do automatic dynamic provisioning if the physical disk isn’t pre-configured. We must define a local StorageClass and manually register PV objects for each node’s physical disk.

1. Creating the Local StorageClass #

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-nvme-sc
provisioner: kubernetes.io/no-provisioner # Mandatory: No external dynamic provisioner
volumeBindingMode: WaitForFirstConsumer # Mandatory: Wait for the scheduler to determine the Pod's node

2. Creating the Local PersistentVolume Statically #

We must specify the NVMe disk directory location on the worker node host and create a node affinity binding so the Kubernetes scheduler knows where this physical disk actually lives.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv-worker-node-1
spec:
  capacity:
    storage: 500Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-nvme-sc
  local:
    path: /mnt/disks/nvme0n1 # The physical NVMe disk mount path on the worker node
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - worker-node-1 # This disk physically ONLY attaches to worker-node-1

Local Storage Trade-offs #

Although it delivers millions of IOPS and near-zero latency performance, we must understand the consequences of using it:

• No Portability: If worker-node-1 suffers motherboard damage, our database Pod can’t move to another node because our physical data is locked on that broken server.
• Replication Responsibility at the Application Level: We must implement data replication architecture at our application software level (e.g. building a PostgreSQL Primary-Replica architecture, Cassandra clusters, or Kafka replication architecture) to anticipate worker node failures.

The Influence of Node Network Bandwidth on Storage #

When using cloud network volumes (like AWS EBS), our volume’s I/O performance isn’t only limited by the volume spec itself, but strictly limited by our worker node VM instance’s network bandwidth.

On cloud providers like AWS, every VM instance (EC2) has an upper limit of dedicated network capacity for storage I/O (EBS Bandwidth).

Let’s look at the following instance bandwidth comparison table:

If we create a high-spec gp3 volume with 12,000 IOPS and 500 MB/s throughput, then attach that volume to a Pod scheduled on a t3.medium worker node, our database performance gets drastically throttled. The t3.medium VM unilaterally throttles our I/O because it runs out of instance network bandwidth allocation.

How to Test Storage Performance Independently #

We must not assume cluster storage performance just by reading cloud provider spec brochures. We must empirically measure real performance using the industry-standard benchmark tool: fio (Flexible I/O Tester).

Let’s create a temporary test Pod manifest that mounts the PVC we want to test:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: benchmark-test-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: secure-database-sc
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: storage-benchmark-agent
  namespace: default
spec:
  restartPolicy: Never
  containers:
    - name: benchmark-container
      image: alpine:latest
      command: ["/bin/sh", "-c", "apk add --no-cache fio && sleep infinity"]
      volumeMounts:
        - name: target-storage
          mountPath: /data
  volumes:
    - name: target-storage
      persistentVolumeClaim:
        claimName: benchmark-test-pvc

Once the Pod is Running, enter the container’s terminal:

kubectl exec -it storage-benchmark-agent -n default -- sh

1. Random Read/Write Test (OLTP Database Simulation) #

Run the following fio command to simulate random database query loads with an 8KB block size:

fio --name=db-simulation \
    --directory=/data \
    --ioengine=libaio \
    --rw=randrw \
    --rwmixread=75 \
    --bs=8k \
    --iodepth=64 \
    --size=5G \
    --direct=1 \
    --runtime=60 \
    --time_based \
    --group_reporting

• --rw=randrw & --rwmixread=75: Simulates a mixed random query load with 75% read and 25% write operations.
• --direct=1: Bypasses the container OS cache (page cache) to ensure we’re purely testing the physical disk performance.

2. Sequential Write Throughput Test (Log / Kafka Simulation) #

Run the following command to measure large sequential file write bandwidth:

fio --name=sequential-write-test \
    --directory=/data \
    --ioengine=libaio \
    --rw=write \
    --bs=1M \
    --iodepth=4 \
    --size=10G \
    --direct=1 \
    --runtime=60 \
    --time_based \
    --group_reporting

Pay attention to the IOPS and BW (Bandwidth/Throughput) sections of the output to learn our cluster storage’s real performance capacity.

Storage Performance Anti-Patterns vs Solutions #

Let’s study some storage performance configuration mistakes (anti-patterns) that often happen in production clusters, along with how to fix them.

Anti-Pattern 1: Using Standard HDD Disk Types for Database Workloads #

We use the default standard magnetic HDD StorageClass to hold PostgreSQL database data to minimize cloud spending.

Wrong Manifest Code (HDD StorageClass with High Latency) #

# DON'T USE THIS FOR PRODUCTION DATABASES
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cheap-hdd-sc
provisioner: ebs.csi.aws.com
# Uses the slow sc1 (Cold HDD) volume type for random queries
parameters:
  type: sc1

Solution Code (Optimized gp3 SSD StorageClass) #

# SOLUTION: GP3 SSD configuration with provisioned IOPS and Throughput
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: optimized-database-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  iops: "5000"       # Increased from the 3000 baseline to 5000 IOPS
  throughput: "250"  # Throughput set to 250 MB/s to support heavy queries
  encrypted: "true"

Anti-Pattern 2: Scheduling High-Performance DB Pods on Small VM Nodes #

We configure a database StatefulSet with a super-fast SSD PVC (10,000 IOPS), but let the database Pod get scheduled on a small VM instance type worker node without dedicated cloud storage bandwidth support.

Wrong Manifest Code (Database Running on a Small Node Without Scheduling Control) #

# DON'T DO THIS: Database vulnerable to throttling at the VM node level
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: high-perf-mysql
  namespace: production
spec:
  replicas: 1
  serviceName: mysql-service
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          # No nodeSelector / affinity: The Pod is free to land on a small t3.medium VM node

Solution Code (Applying NodeAffinity to an Optimized Node Pool) #

We must group our worker nodes into a dedicated node pool containing high-performance VMs (EBS-optimized, e.g. m5.xlarge instance types and above), then label those nodes.

# SOLUTION: Locking database Pod placement to a dedicated high-performance node pool
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: high-perf-mysql
  namespace: production
spec:
  replicas: 1
  serviceName: mysql-service
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type-group
                    operator: In
                    values:
                      - storage-optimized-nodes # Only schedule the Pod in the m5.xlarge/m5.2xlarge node pool
      containers:
        - name: mysql
          image: mysql:8.0
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
            limits:
              cpu: "4"
              memory: "8Gi"
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: optimized-database-gp3-sc
        resources:
          requests:
            storage: 100Gi

Summary #

• Identify your application’s I/O profile: Choose storage allocation based on the application’s need priorities. Transactional databases need high IOPS and low latency, while Kafka/Logs need large throughput.
• Use Local Persistent Volumes for zero latency: For workloads highly sensitive to query latency, local NVMe is the best solution, provided data replication management is handled at the application level.
• I/O bandwidth is tied to VM node size: Don’t attach high-spec StorageClass disks to small VM instance nodes to avoid unilateral performance throttling by the cloud infrastructure.
• Run empirical benchmarks with fio: Always do read/write tests using the fio utility directly inside cluster Pods to measure real IOPS and bandwidth before releasing systems to production.
• Apply Node Affinity for Pod placement: Make sure high-performance database Pods have their placement locked to node pools containing high-spec VMs optimized for storage operations.

← Previous: Storage Anti-Patterns   Next: Kubernetes Network Model →