Databases in Kubernetes #

Running databases inside Kubernetes is one of the topics that most often sparks fierce debate among system engineers. On one side, there’s a group aggressively arguing that Kubernetes was designed purely for stateless workloads and running databases on it is an instant recipe for data loss disaster. On the other side, many large technology companies successfully run thousands of database clusters reliably on Kubernetes.

In reality, both arguments aren’t wrong — the operational context and the team’s expertise level are what make the difference. Kubernetes has evolved far beyond a stateless container orchestrator into a very powerful cloud-native orchestration platform for stateful workloads, thanks to the StatefulSet API and the Operator Pattern. This article objectively dissects database characteristics compared to stateless applications, feasibility guides for deploying in-cluster databases, production single-StatefulSet manifest writing, the importance of Headless Services, and advanced operational automation using Operators.

Database vs Stateless Application Characteristics #

Before deciding to put a database into a Kubernetes cluster, we must understand the fundamental runtime behavior differences between stateless applications and database engines:

Given database characteristics that are highly sensitive to identity and connectivity changes, the kube-scheduler and Controller Manager apply much stricter safety rules on StatefulSets than on Deployments.

Decision Framework: When Is a Database in K8s Worth It? #

Running databases in Kubernetes carries high long-term operational maintenance consequences (day-2 operations). Use the following decision matrix before going further:

1. Criteria for Running a Database in Kubernetes: #

• High Portability Needs: We need a system that runs exactly the same on on-premise, AWS, GCP, and Azure environments without being tied to a specific cloud provider API.
• Availability of a Mature Official Operator: Our database has an active, battle-tested Operator (like Strimzi for Apache Kafka or CloudNativePG for Postgres).
• Development and Staging Environments: Running databases in Kubernetes for non-production environments makes a lot of sense to save on expensive managed cloud database rental costs.
• Small-Scale Microservices: Micro databases dedicated to one service with low-to-medium traffic.

2. Criteria for Preferring a Managed DB Service (RDS/Cloud SQL/Atlas): #

• Downtime Is Very Expensive: Our business can’t tolerate more than a few minutes of database downtime per year (99.99% SLA).
• Team Expertise Scarcity: Our DevOps/SRE team lacks deep expertise in distributed database replication internals, quorum election, and filesystem disaster recovery.
• Limited Operational Bandwidth: We want to fully hand over routine backup responsibilities, OS security patching, and multi-zone failover to the cloud provider.

The Basic Pattern: A Production Single PostgreSQL StatefulSet Manifest #

For simple non-clustered (single-instance) databases used for internal needs, we can put together a reliable StatefulSet manifest.

The most critical thing is Health Probe configuration (Liveness & Readiness). A database doing data recovery (WAL replay) after a crash must not be considered failed by the Liveness Probe; if the Kubelet kills it mid-recovery, our database enters a corruption loop.

Here’s a single PostgreSQL StatefulSet manifest with safe probe configuration:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-db
  namespace: database
spec:
  serviceName: "postgres-headless"   # Points to the Headless Service
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      securityContext:
        fsGroup: 999                 # postgres user ID for volume permissions
      containers:
      - name: postgres
        image: postgres:15.4-alpine
        env:
        - name: POSTGRES_DB
          value: "production_db"
        - name: POSTGRES_USER
          value: "pg_admin"
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-credentials
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
          name: dbport
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
        volumeMounts:
        - name: db-data-vol
          mountPath: /var/lib/postgresql/data
        # Liveness probe uses pg_isready with a loose startup time buffer:
        livenessProbe:
          exec:
            command:
            - pg_isready
            - -U
            - pg_admin
          initialDelaySeconds: 60    # Gives 1 minute for WAL replay/startup
          periodSeconds: 10
          timeoutSeconds: 5
        # Readiness probe ensures the db is ready to accept SQL traffic:
        readinessProbe:
          exec:
            command:
            - pg_isready
            - -U
            - pg_admin
          initialDelaySeconds: 15
          periodSeconds: 5
  volumeClaimTemplates:
  - metadata:
      name: db-data-vol
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "premium-ssd-sc" # High IOPS SSD StorageClass
      resources:
        requests:
          storage: 50Gi

The Importance of a Headless Service for Database Networking #

A StatefulSet must be paired with a Headless Service. A Headless Service is a regular Kubernetes Service defined with the clusterIP: None parameter.

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: database
spec:
  clusterIP: None                    # The key parameter that makes it Headless
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

Why Is the Headless Service So Vital? #

1. Direct IP Resolution: A regular Service has a virtual ClusterIP acting as a random Load Balancer. That doesn’t suit databases. Clients (our applications) must be able to hit a specific database Pod IP (e.g. hitting the master node for writes, and replica nodes for reads).
2. Stable DNS: With a Headless Service, Kubernetes CoreDNS registers a unique FQDN address for each StatefulSet Pod: $$\text{FQDN} = \text{postgres-db-0.postgres-headless.database.svc.cluster.local}$$ Even if Pod postgres-db-0 restarts and gets a new internal IP, the DNS address above is guaranteed to stay the same, preventing connection failures in our application configuration.

[!CAUTION] Never expose our internal database to the internet using a LoadBalancer Service or NodePort. This is a very dangerous security hole. Accessing the database from outside the cluster for debugging needs should always use a secure path like a cluster VPN or the secure port-forwarding command: kubectl port-forward statefulset/postgres-db 5432:5432 -n database.

The Operator Pattern: Automating Complex (Day-2) Operations #

Deploying a single-instance database with a regular StatefulSet is very easy. However, when we need to run multi-node database clusters with active replication, scheduled automatic backups to S3, Point-in-Time Recovery (PITR), and automatic failover when the Master dies, a standard StatefulSet isn’t enough. We need an Operator.

An Operator is an extension of human intelligence written into code. Operators listen to custom database Custom Resource Definitions (CRDs) and run reconciliation loops to ensure the cluster runs per spec.

Here’s a diagram of the automatic reconciliation workflow performed by a Database Operator:

flowchart TD
    CRD_Deploy["Developer Deploys CRD: apiVersion: cloudnative-pg.io/v1\nkind: Cluster (instances: 3)"] --> OperatorLoop["Operator Reconciliation Loop Active"]
    
    subgraph OperatorLogic["Operator Automation Logic"]
        OperatorLoop --> DetectState["Detect Current State: 0 Running Instances"]
        DetectState --> CreateMaster["1. Create master-0 Pod & master-0 PVC"]
        CreateMaster --> WaitMasterReady["Wait for master-0 to Be Ready"]
        
        WaitMasterReady --> CreateSlaves["2. Create replica-1 & replica-2 Pods"]
        CreateSlaves --> BootstrapReplication["3. Run rsync/pg_basebackup from master-0 to replica-1 & 2"]
        BootstrapReplication --> StartSync["4. Enable Streaming Replication (Sync)"]
    end
    
    StartSync --> MonitorHA["Continuously Monitor Cluster Health"]
    MonitorHA --> DetectMasterCrash{"Did the Master\nPod master-0 Crash?"}
    
    DetectMasterCrash -- "No" --> MonitorHA
    DetectMasterCrash -- "Yes" --> FailoverAction["Pick replica-1 as the New Master\nUpdate DNS Config & Promote to Primary"]
    FailoverAction --> MonitorHA

Popular Database Operators in the Industry: #

• CloudNativePG (PostgreSQL): One of the best PostgreSQL operators, designed natively for Kubernetes without complex wrappers like Patroni. Very efficient at managing backups to Object Storage (S3/GCS) and streaming replication.
• Zalando Postgres Operator: Battle-tested at giant production scale by the Zalando team. Uses Spilo (PostgreSQL + Patroni) for high availability (HA) consensus.
• Strimzi (Apache Kafka): Automates Kafka and ZooKeeper cluster management, user management, and declarative topic creation via the Kubernetes API.
• Percona Operator for MySQL / MongoDB: Very reliable for multi-master cluster setups and distributed replication with integrated backup features.

Production Checklist Before Launching a Database in Kubernetes #

Before pressing the deploy button for a database into a Kubernetes production environment, make sure we’ve checked the entire safety list below:

1. Storage Hardening #

• Retain Reclaim Policy: Make sure the StorageClass or manual PV uses the Retain policy so data isn’t lost if a PVC is accidentally deleted.
• WaitForFirstConsumer: Minimizes scheduling errors from cloud zone mismatches.
• High IOPS: Use a StorageClass supporting SSD disk types (gp3/pd-ssd) with a minimum of 3000 dedicated IOPS.
• Volume Expansion: Make sure the SC has the allowVolumeExpansion: true parameter.

2. Network and Secret Security #

• Secret Injection: Database credentials must be hidden in Secrets, not ConfigMaps or static env vars.
• NetworkPolicy: Restrict incoming access to the database. Only backend Pods with specific labels may communicate with the database port.
• Non-Root Execution: Make sure the database container runs as a non-root user (e.g. user ID 999 owned by postgres) with the runAsNonRoot: true configuration in the SecurityContext.

3. Service Resilience (High Availability) #

• Anti-Affinity Rules: Apply podAntiAffinity rules so database master and slave Pods are guaranteed to be scheduled on different physical worker nodes to anticipate single-node death.
• Connection Pooler: Use a connection pooler tool (like PgBouncer) to prevent database connection slot exhaustion from backend Pod autoscaling.

4. Observability & Alerting #

• Exporter Metrics: Install a database exporter (like postgres_exporter) to ship metrics to Prometheus.
• Disk & Connection Alerting: Set critical alerts if volume disk usage > 80% or connection usage percentage > 90%.

Database Management Anti-Patterns in Kubernetes #

Here are three fatal configuration mistakes that often paralyze production clusters when running databases:

Anti-Pattern 1: Using a Regular (Stateless) Deployment for Replicated Databases #

Deploying a Master-Slave relational database using a standard Deployment object because it’s considered easier to configure.

ANTI-PATTERN: Deployment with replicas: 3 for a PostgreSQL Cluster
// WHAT WE DO:
- Write a Deployment manifest with replicas: 3 and attach it to one shared PVC.

// THE CONSEQUENCES IN PRODUCTION:
- Write Collisions (Data Corruption): Three Pods try to write to the same data folder simultaneously.
  Because relational databases aren't designed for multi-writer without an external lock manager,
  the database filesystem immediately corrupts completely (*corrupted filesystem*).
- Random Identity: Pods get random names (like `db-app-8fbc10-xyz`). Application clients can't
  distinguish master from slave because IPs and pod names change on every restart.

✓ THE RIGHT SOLUTION:
- Always use the **StatefulSet** object for databases. StatefulSets guarantee ordinal numbering
  and exclusive PersistentVolume allocation that doesn't overlap between Pod replicas.

Anti-Pattern 2: Setting Liveness Probes Too Aggressively Without a Startup Buffer #

Configuring a database Liveness Probe with strict timeout parameters to detect service failure as fast as possible.

ANTI-PATTERN: livenessProbe with initialDelaySeconds: 5 and failureThreshold: 2
// WHAT WE DO:
- Our database suffers a sudden death because the node restarted. When it comes back, the database must read
  the last transaction files (WAL Replay) to repair table integrity. This process takes 30 seconds.
- However, we set `initialDelaySeconds: 5`.

// THE CONSEQUENCES IN PRODUCTION:
- Death Loop (Crash Loop): Five seconds after starting, the Kubelet begins probing.
  Because the database is still busy processing WAL recovery, it doesn't answer the probe ping.
- The Kubelet considers the database dead and immediately force-kills the container (restart).
- The database container starts again, repeats the recovery process from scratch, and gets killed again after 5 seconds.
  Our database enters a death loop and can never start successfully.

✓ THE RIGHT SOLUTION:
- Give a realistic `initialDelaySeconds` value (e.g. 60 or 90 seconds) specifically for databases.
- As an alternative, use a **Startup Probe** first to lock the Liveness Probe so it doesn't run
  before the database truly completes all its internal recovery processes.

Anti-Pattern 3: Connecting the Database to a Low-Performance Default StorageClass #

Ignoring the StorageClass specification on the database’s volumeClaimTemplates, so it automatically uses the cluster’s default StorageClass.

ANTI-PATTERN: volumeClaimTemplates Without a storageClassName Spec
// WHAT WE DO:
- Leave the `storageClassName` property empty on the database claim template.

// THE CONSEQUENCES IN PRODUCTION:
- High I/O Latency: Kubernetes creates the disk automatically using the cluster's default StorageClass
  (which usually uses standard mechanical HDD disks for cost savings).
- Database transactions needing high disk write speed suffer severe I/O queueing.
  Our backend applications start producing HTTP 504 Gateway Timeout errors at peak traffic.

✓ THE RIGHT SOLUTION:
- Always explicitly specify a dedicated high-performance SSD StorageClass (like `premium-ssd-sc`)
  on our database StatefulSet's `volumeClaimTemplates` block.

Summary #

• StatefulSet Is the Foundation — Always use the StatefulSet object for databases to guarantee network identity stability and exclusive per-Pod storage allocation.
• Headless Service Is Mandatory — Connect the StatefulSet to a Headless Service (clusterIP: None) so CoreDNS can register stable FQDNs for each database Pod.
• Loosen Liveness Probes — Give enough time buffer (initialDelaySeconds) or use Startup Probes to give the database room to perform WAL recovery.
• Use Operators for HA — Apply official Operators (like CloudNativePG or Strimzi) to manage complex database clusters with automatic failover and automated backups.
• Bypass to Managed DBs — Don’t hesitate to choose managed cloud database services (RDS/Cloud SQL) if your team lacks day-2 database operations specialization in Kubernetes.
• Apply podAntiAffinity — Protect database clusters from physical node failure by explicitly spreading Pod replicas across different worker nodes.
• Don’t Expose the Database — Isolate database ports inside the cluster network using NetworkPolicy and never use a LoadBalancer Service to expose them publicly.

← Previous: StorageClass   Next: StatefulSet + PVC →