StatefulSet #

In distributed system architecture, we recognize two workload categories: Stateless and Stateful. Stateless applications — like API Gateways, frontends, or regular microservices — can be easily killed, moved, or duplicated at any time because they don’t store valuable local data. However, stateful applications — like PostgreSQL, MySQL, Redis caches, Kafka message queues, or Elasticsearch clusters — need much more complex handling. They must always remember their identity, where their data is stored, and in what order they should communicate.

To manage these stateful applications safely, Kubernetes provides a special controller called StatefulSet. A StatefulSet acts as a guardian giving each Pod replica a unique identity, stable networking, and persistent storage permanently bound to it. Understanding how StatefulSet works in depth helps us avoid catastrophic data loss and makes database maintenance on Kubernetes much easier.

Fundamental Differences from Deployment #

Many beginner DevOps teams try to deploy databases using Deployment objects because they’re more familiar. That’s a fatal mistake.

Here’s an architectural comparison table distinguishing how Deployment and StatefulSet handle Pods:

A StatefulSet treats every Pod as a unique individual that can’t replace each other (pet model), while a Deployment treats Pods as identical, disposable herd animals (cattle model).

StatefulSet Manifest Structure in Depth #

To build a resilient database cluster, we must write the StatefulSet manifest including the volumeClaimTemplates declaration block and link it to a Headless Service.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb
  namespace: database
spec:
  serviceName: "mariadb-headless"       # The Headless Service name is mandatory
  replicas: 3
  podManagementPolicy: OrderedReady      # OrderedReady (Default) | Parallel
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb:10.11
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: db-data
          mountPath: /var/lib/mysql
  # Each Pod automatically gets its own storage claim
  volumeClaimTemplates:
  - metadata:
      name: db-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "premium-rwo-ssd"
      resources:
        requests:
          storage: 20Gi

Key Parameter Explanations: #

• serviceName: The connecting key pointing to the Headless Service object name. Without this field matching, individual Pod DNS resolution won’t work.
• volumeClaimTemplates: The automatic PersistentVolumeClaim (PVC) creation template. If we request 3 replicas, the StatefulSet Controller dynamically creates 3 independent PVCs in the API Server: db-data-mariadb-0, db-data-mariadb-1, and db-data-mariadb-2.

Stable Identity: Ordinal Naming Scheme and DNS Resolution #

Every Pod created by a StatefulSet gets an ordinal index number from 0 to N-1 (where N is the replica count). This number permanently sticks to the Pod’s physical and network identity.

Name and Hostname Stability #

If Pod mariadb-1 crashes or its Worker Node dies, the Scheduler reschedules that Pod to another healthy node.

• The Name Stays the Same: The new Pod that starts still has the name mariadb-1.
• Storage Stays Bound: The Kubelet tracks PVC db-data-mariadb-1 and reattaches it to the new mariadb-1 Pod. Our database data won’t get mixed up with data belonging to mariadb-0 or mariadb-2.

Unique DNS FQDN Format #

With the help of a Headless Service, every Pod has a stable Fully Qualified Domain Name (FQDN) inside the Kubernetes CoreDNS internal network:

$$\text{DNS Format} = \text{{pod-name}}.\text{{service-name}}.\text{{namespace}}.svc.cluster.local$$

MariaDB cluster with 3 replicas has these internal DNS addresses:
  - mariadb-0.mariadb-headless.database.svc.cluster.local
  - mariadb-1.mariadb-headless.database.svc.cluster.local
  - mariadb-2.mariadb-headless.database.svc.cluster.local

These domain names never change even though the dynamic Pod IPs behind them change due to restarts. This makes database cluster synchronization configuration easy (like designating the master node at index 0 and slave nodes at indices 1 and 2).

The Vital Role of the Headless Service #

A StatefulSet must be paired with a Headless Service. A Headless Service is a regular Kubernetes Service, but defined with the clusterIP: None parameter.

apiVersion: v1
kind: Service
metadata:
  name: mariadb-headless # Must MATCH the spec.serviceName in the StatefulSet
  namespace: database
spec:
  clusterIP: None       # The key parameter that makes it Headless
  selector:
    app: mariadb
  ports:
  - port: 3306
    targetPort: 3306

Why Must It Be clusterIP: None? #

• No Virtual Load Balancer: A regular Service has a single virtual IP (ClusterIP) doing random (round-robin) load balancing to all Pods behind it. This pattern doesn’t suit databases because clients must know specifically whether they’re hitting the Master Node (for writes) or a Slave Node (for reads).
• Direct IP Resolution: A Headless Service has no ClusterIP. When an application queries DNS for mariadb-headless, CoreDNS doesn’t return one virtual IP; it returns a list of direct IPs from all healthy connected Pods. Application clients can choose which Pod IP to hit directly.

Ordered Startup and Shutdown Operations #

To prevent data skew during distributed cluster formation, the StatefulSet Controller enforces strict execution ordering rules.

Here’s a visualization of the ordered scale-down process on a StatefulSet:

flowchart TD
    StartScaleDown["Start Scale-Down Command (3 Replicas ➔ 1 Replica)"] --> SelectHigh["1. Pick the highest-index Pod: mariadb-2"]
    SelectHigh --> TerminateHigh["2. Send the termination signal (SIGTERM) to mariadb-2"]
    
    TerminateHigh --> WaitHigh{"3. Is mariadb-2\ncompletely dead?"}
    WaitHigh -- "Not yet" --> WaitHigh
    WaitHigh -- "Yes" --> SelectNext["4. Pick the next highest-index Pod: mariadb-1"]
    
    SelectNext --> TerminateNext["5. Send the termination signal to mariadb-1"]
    
    TerminateNext --> WaitNext{"6. Is mariadb-1\ndead?"}
    WaitNext -- "Not yet" --> WaitNext
    WaitNext -- "Yes" --> Stop["7. Scale-down complete. Only mariadb-0 is running."]

OrderedReady vs Parallel Management #

By default, StatefulSet applies the podManagementPolicy: OrderedReady policy:

• OrderedReady: The next Pod isn’t created until the previous Pod is declared truly Ready. This is crucial for clusters like Apache ZooKeeper that need gradual bootstrap quorum.
• Parallel: Removes all ordering constraints. All Pods start or stop in parallel simultaneously. This policy is perfect when we only need name and PVC stability but want cluster provisioning to finish as fast as possible (e.g. Apache Cassandra clusters).

Update Policy and the Partition Canary Feature #

StatefulSet supports the RollingUpdate strategy, which processes updates from the highest index to the lowest sequentially. Additionally, StatefulSet has an advanced feature called Partition.

The partition property lets us do Canary releases at the database level. Imagine a database cluster with 5 replicas (indices 0 to 4) where we want to test a new version only on indices 3 and 4:

spec:
  replicas: 5
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 3 # Only update Pods with index >= 3 to the new version

With the configuration above:

• If we update the StatefulSet image to a new version, only mariadb-3 and mariadb-4 get restarted and updated to the new version.
• Pods mariadb-0, mariadb-1, and mariadb-2 are guaranteed to keep running the old version.
• This gives DevOps teams room to test the new version’s performance stability on one or two slave replicas first before rolling out the release fully to the main (master) database.

Volume Retention Policy (PVC Retention) #

An important StatefulSet behavior every cluster administrator must understand: When a StatefulSet is deleted or scaled down, the PVC objects are never deleted along with it.

• Disaster Prevention: This is a deliberate Kubernetes design decision to protect our valuable data. If we accidentally delete a StatefulSet, the data on disk stays safe. When we recreate a StatefulSet with the same name, the old data reconnects automatically.
• Manual Cleanup: To truly discard data, we must manually delete the PVCs after the StatefulSet is gone:

  kubectl delete pvc data-mariadb-0 data-mariadb-1
  

The Modern Automatic Retention Feature #

In modern Kubernetes versions, we can control this PVC retention behavior declaratively with the persistentVolumeClaimRetentionPolicy block:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete # Automatically delete PVCs if the StatefulSet object is deleted
    whenScaled: Retain  # Keep the PVCs if only scaling down

Node Failure Handling and the “At-Most-One” Rule #

One of the most misunderstood StatefulSet behaviors is node failure handling. On Deployment (stateless) objects, if a Worker Node dies (becomes NotReady), Kubernetes detects the lost communication (after a certain duration, usually controlled by kube-controller-manager parameters) and automatically recreates the Pod on another healthy node.

However, this behavior is very different on StatefulSets. To maintain strict data consistency and prevent data corruption from double writes, StatefulSet enforces the At-Most-One Pod Semantics rule.

Why Is At-Most-One So Important? #

Imagine running a PostgreSQL database cluster with an active replication topology. Pod db-0 acts as the Master Node writing data to external disk (cloud storage). Suddenly, the node hosting db-0 experiences a mild network partition. The Control Plane can’t reach that node, so it shows NotReady status in the API Server’s eyes. However, the node is actually still running, and the database process inside is still actively writing data.

If Kubernetes naively created a new db-0 Pod on another node and connected it to the same storage volume, we’d have two different database processes writing to the same disk simultaneously. That’s a major disaster called split-brain, which will certainly corrupt the database tables and destroy all our data.

Pod Lifecycle When a Node Fails #

When a Node dies, the StatefulSet Pods on it get marked with Terminating or Unknown status by the control plane. However:

1. The Pod Won’t Be Automatically Deleted: The kubelet on the dead node can’t confirm to the API Server that the containers inside have truly stopped.
2. No New Pod Will Be Created: The StatefulSet Controller refuses to create a replacement Pod with the same ordinal name (e.g. db-0) on another node until the old Pod is truly dead. Our Pod stays stuck in Terminating status forever.

Safe Administrator Handling Steps #

To recover this state in production, we must not force-delete the Pod without first doing physical verification. Here are the safe steps to follow:

1. Verify the Physical Node Condition: Make sure the old node is truly dead (permanent hardware failure) or has been physically disconnected from the storage network.
2. Do an Evacuation / Force Delete: If and only if we’re certain the old node will never write to the storage volume again, we can force-delete the Pod with:

   kubectl delete pod db-0 --grace-period=0 --force -n database
   

   After this command runs, the API Server deletes the old Pod entry from etcd, and the StatefulSet Controller immediately schedules a new db-0 Pod on another node safely.

Anti-Patterns in StatefulSet Management #

Here are two fatal mistakes that often corrupt data or paralyze database clusters:

Anti-Pattern 1: Skipping the Headless Service or Mapping serviceName Wrong #

A service name configuration mistake that breaks the cluster’s DNS resolution communication.

ANTI-PATTERN: Writing serviceName: "mariadb-service" But Creating a Service Without clusterIP: None
// WHAT WE DO:
- Write `serviceName: "mariadb-headless"` in the StatefulSet manifest.
- But forget to create a Service object named `mariadb-headless`,
  or create that Service as a standard Service (with a virtual ClusterIP).

// THE CONSEQUENCES IN PRODUCTION:
- DNS Breakage: Individual Pod addresses like `mariadb-0.mariadb-headless` fail to resolve via CoreDNS.
- Replication Failure: The database cluster fails to sync master-slave data
  because the database nodes can't find each other internally, triggering database split-brain status.

✓ THE RIGHT SOLUTION:
- Always ensure a companion Service object with `clusterIP: None` exists, with a name
  exactly matching the `spec.serviceName` property written in the StatefulSet manifest.
- Verify DNS name resolution from inside the cluster with a diagnostic command:
  `nslookup mariadb-0.mariadb-headless.database.svc.cluster.local`

Anti-Pattern 2: Using Slow/Non-SSD Storage Classes for Stateful Databases #

Choosing a cheap standard StorageClass to cut infrastructure costs.

ANTI-PATTERN: Using storageClassName: "standard-hdd" for a PostgreSQL Database
// WHAT WE DO:
- Deploy a production database cluster with regular mechanical hard disk (HDD) PVC types.

// THE CONSEQUENCES IN PRODUCTION:
- I/O Bottleneck: Stateful databases perform high-I/O write transactions. Slow HDDs trigger write queue buildup.
- Crash Loops: The database process often fails the liveness probe due to I/O hangs,
  making the Kubelet restart the Pod repeatedly, damaging database table integrity.

✓ THE RIGHT SOLUTION:
- Always use SSD storage classes with guaranteed performance (e.g. `premium-rwo-ssd` on GCP or `gp3/io2` on AWS).
- Configure the `volumeClaimTemplates` parameters to always point to that SSD `storageClassName`.

Summary #

• Unique Stable Identity — StatefulSet gives permanent ordinal names (db-0, db-1) and dedicated PersistentVolume allocation to each Pod replica, keeping data stable across restarts.
• Headless Service Is Mandatory — Without defining a Headless Service (clusterIP: None), per-Pod DNS domain name resolution won’t work in the cluster.
• Ordered Lifecycle Operations — Pod formation runs in order from the smallest index (0 ➔ 1 ➔ 2), while shutdown runs in reverse from the largest index (2 ➔ 1 ➔ 0) to keep data replication safe.
• Protected PVC Retention — By default, Kubernetes doesn’t delete PVCs when a StatefulSet dies to protect data; use the modern retention policy or delete PVCs manually if the data should be discarded.
• Canary Releases via Partition — Leverage the rollingUpdate.partition feature on StatefulSets to test a new image version on just the highest-index Pods before rolling out to the whole database.
• Avoid Deployments for DBs — Don’t use regular Deployments for replicated databases to prevent storage volume write collisions and loss of network identity state.
• Use SSD Storage — Always bind volumeClaimTemplates to an SSD-based StorageClass to guarantee optimal database I/O transaction speed.

← Previous: Deployment   Next: DaemonSet →