Disaster Recovery #

In IT infrastructure operations, the only thing worse than not having data backups is assuming our data backups work without ever testing the restoration process. In the cloud-native architecture era, Kubernetes clusters are vulnerable to various disaster scenarios, from physical hardware damage in cloud data centers, ransomware cyber attacks damaging etcd, to human errors accidentally deleting main production namespaces. Disaster Recovery (DR) isn’t just a data copying activity, but a Business Continuity plan carefully designed so systems can recover within agreed time tolerance thresholds. This article discusses cluster backup object classification, Velero tool implementation, consistent database backup strategies, multi-cluster recovery architecture comparisons, and robust restoration testing simulations.

What Must Be Backed Up? #

A Kubernetes cluster has two state categories that must be backed up separately because they have very different recovery methods:

+-------------------------------------------------------------------------+
|                        CLUSTER BACKUP CATEGORIES                        |
+---------------------------------------+---------------------------------+
|   1. CLUSTER STATE (METADATA)         |   2. WORKLOAD DATA (PERSISTENT) |
|   - Object configurations in etcd     |   - Real data in Persistent Volumes |
|   - Deployments, Services, RBAC, Secrets | - Database records, user uploads  |
|   - Can be backed up via Git (GitOps) |   - Not in Git (Static)            |
|   - Solution: Declarative Git repos   |   - Solution: Volume Snapshots & Dumps |
+---------------------------------------+---------------------------------+

1. Cluster Metadata Guarantees (Cluster State) #

All object manifests defining our cluster (like Deployment, Service, ConfigMap, Secret, NetworkPolicy, and RBAC configurations) are stored in the Control Plane’s internal etcd database.

If we apply the GitOps principle disciplinedly using tools like ArgoCD or Flux CD, our Git repository automatically acts as the best Cluster State backup. If the cluster is totally destroyed, we just build a new empty cluster, then connect it to our Git repository to automatically reinstall all objects.

2. Application Data Guarantees (Persistent Workload Data) #

GitOps can’t back up dynamic data stored by applications on storage disks (like PostgreSQL databases, RabbitMQ message queues, or user upload file folders). This data is physically stored in cloud storage providers through the PersistentVolume (PV) abstraction.

These PV storages are what we must back up using Volume Snapshot mechanisms and periodic native database export processes.

Velero: The Industry Backup and Restore Standard #

Velero is an open-source tool managed by VMware specifically designed for natively backing up and restoring Kubernetes metadata objects and Persistent Volumes. Velero interacts with the Kubernetes API Server to take object snapshots, compress them, then store them in Object Storage (like AWS S3, Google Cloud Storage, or Harbor) using related cloud plugins.

flowchart TD
    Admin["Administrator (velero CLI)"] -->|"1. Send a Backup Custom Resource"| K8sAPI["Kubernetes API Server"]
    K8sAPI -->|"2. Detect the New CR"| VeleroPod["Velero Controller Pod"]
    
    subgraph ClusterBackup["Velero internal process"]
        direction TB
        VeleroPod -->|"3. Query Object Metadata"| K8sAPI
        VeleroPod -->|"4. Request Disk Snapshots via CSI"| CSIDriver["CSI Storage Driver"]
    end
    
    VeleroPod -->|"5. Upload the YAML Archive (.tar.gz)"| S3["Object Storage (S3 / GCS)"]
    CSIDriver -->|"6. Create Disk Block Snapshots"| CloudSnapshot["Cloud Provider Disk Snapshots"]

1. Production-Level Velero Installation (AWS S3 Backend) #

To install Velero, we must provide access credentials for the cluster-external S3 storage bucket.

# Install the Velero controller binary into the cluster
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket production-k8s-backups \
  --backup-location-config region=ap-southeast-1 \
  --snapshot-location-config region=ap-southeast-1 \
  --secret-file ./credentials-velero # An IAM credentials file with S3 & EC2 Snapshot permissions

2. Daily Backup Command Management #

Once installed, we can automate backup creation using declarative commands:

# 1. Create a one-time backup for the 'prod-apps' namespace
velero backup create prod-apps-backup-20260617 \
  --include-namespaces prod-apps

# 2. Create an automatic backup schedule every day at 02:00 WIB early morning
velero schedule create daily-prod-backup \
  --schedule="0 2 * * *" \
  --include-namespaces prod-apps \
  --ttl 720h0m0s # Set the backup expiration period to 30 days (Time-To-Live)

# 3. Check the backup result health status
velero backup get
velero backup describe prod-apps-backup-20260617

The Velero Restore Mechanism and Isolation Testing #

Letting backup files pile up in S3 without ever trying to restore them is a big mistake. We must get teams used to doing periodic restoration testing in isolated environments.

# 1. Restore the entire namespace from an existing backup file
velero restore create \
  --from-backup prod-apps-backup-20260617 \
  --include-namespaces prod-apps

# 2. IMPORTANT: Restore to a different Namespace for testing (Namespace Mapping)
# This command maps the 'prod-apps' namespace contents from the backup to the new 'prod-restore-test' namespace
velero restore create \
  --from-backup prod-apps-backup-20260617 \
  --namespace-mappings prod-apps:prod-restore-test

# 3. Monitor the object and disk volume recovery progress
velero restore get
velero restore describe <restore-name>

Consistent Database Backups (Database-Native Dumps) #

Although Velero is very reliable for backing up PV disks using cloud provider snapshots, the block-level snapshot method has critical weaknesses when used for active relational databases.

If a snapshot is taken exactly while the database engine is doing large data write operations to disk (active transactions), the data stored in the snapshot risks being in an inconsistent state (crash inconsistent state). When restored, the database can suffer table corruption or lose unfinished transaction data (dirty writes).

Solution: Database-Native Dump CronJobs #

To guarantee database data consistency, we must run data export processes (database dumps) using official database engine utilities (like pg_dump for PostgreSQL). This process reads data directly from database memory safely and structurally.

Here’s a production-level Kubernetes CronJob manifest for safely backing up a PostgreSQL database to an S3 bucket using IRSA (IAM Roles for Service Accounts) authorization without storing AWS passwords in the manifest:

# File: k8s/production-postgres-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-consistent-backup
  namespace: prod-apps
spec:
  schedule: "0 1 * * *" # Run every day at 01:00 WIB early morning (during quiet traffic)
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          # Uses a ServiceAccount bound to an S3 IAM Role (IRSA)
          serviceAccountName: db-backup-sa 
          restartPolicy: OnFailure
          containers:
          - name: pg-backup-agent
            image: postgres:15-alpine
            command:
            - /bin/sh
            - -c
            - |
              set -e # Stop execution if any command fails
              TIMESTAMP=$(date +%Y%m%d_%H%M%S)
              BACKUP_FILE="db_backup_${TIMESTAMP}.sql.gz"
              
              echo "Starting the consistent pg_dump process..."
              # Take an online database dump without locking tables (non-blocking)
              PGPASSWORD=$DB_PASSWORD pg_dump \
                -h $DB_HOST -U $DB_USER -d $DB_NAME \
                | gzip > /tmp/${BACKUP_FILE}
              
              echo "Dump process finished. Downloading the AWS CLI for upload..."
              # Leverage the minimal CLI binary for the upload
              apk add --no-cache aws-cli
              
              echo "Uploading the file to the AWS S3 bucket..."
              aws s3 cp /tmp/${BACKUP_FILE} s3://production-db-backups/postgres/${BACKUP_FILE}
              
              echo "Upload process succeeded: ${BACKUP_FILE}"
              
              # Automatic ROTATION: Delete backups older than 7 days in S3
              echo "Running old backup file rotation..."
              aws s3 ls s3://production-db-backups/postgres/ \
                | awk '{print $4}' \
                | sort -r | tail -n +8 \
                | xargs -I {} aws s3 rm s3://production-db-backups/postgres/{}              
            env:
            - name: DB_HOST
              value: "postgres-primary-service"
            - name: DB_NAME
              value: "banking_production"
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: password

Multi-Cluster Disaster Recovery Strategies #

For critical financial transaction or e-commerce systems that can’t tolerate downtime failures, relying on a single cluster in one cloud provider region is a fatal mistake. If that region suffers natural disasters or submarine fiber optic cable disruptions, our entire system dies.

We must apply one of the following three multi-cluster recovery strategies based on business needs:

flowchart TD
    User["User Traffic (HTTP requests)"] --> GlobalDNS["Global DNS / Traffic Manager (Route53 / Cloudflare)"]
    GlobalDNS -->|"Main Route (Active)"| ClusterPrimary["Main Cluster (Region A)"]
    GlobalDNS -. "Switch Routes on Failure (Standby)" .-> ClusterDR["Disaster Cluster (Region B - Standby)"]
    
    subgraph RegionA["Region A (Primary)"]
        ClusterPrimary --> DBPrimary["Main Database (Primary)"]
    end
    
    subgraph RegionB["Region B (Disaster)"]
        ClusterDR --> DBStandby["Replication Database (Replica)"]
    end
    
    DBPrimary -. "Synchronous/Asynchronous Data Replication" .-> DBStandby
    
    ClusterPrimary -. "Status: Unhealthy (API Server Timeout)" .-> GlobalDNS

1. Active-Passive (Failover Standby) #

Mechanism: The Main Cluster (Region A) serves 100% of transaction traffic. The Disaster Cluster (Region B) is in standby condition. All configurations are identically deployed on both clusters. Database data is asynchronously replicated from Region A to Region B.
Failover: If Region A goes down, Global DNS (like AWS Route53 or Cloudflare) detects the failure through health checks and automatically redirects network routes to Region B.
RTO: 5–15 minutes (depending on DNS propagation duration and the database replica-to-primary promotion process).

2. Active-Active (Multi-Region Active) #

Mechanism: Both clusters in Region A and Region B serve transaction traffic simultaneously (e.g. 50% / 50% via geographic routing).
Advantage: RTO approaches 0 seconds because if one region dies, traffic is instantly redirected to the healthy region without delay.
Challenge: Multi-master database synchronization complexity (data consistency) is very high and requires 2x rental costs because both clusters must fully operate.

3. GitOps Cluster Recreation (Cold Standby) #

Mechanism: We don’t rent a second backup cluster to save costs.
Operations: When disasters happen, SRE teams trigger infrastructure automation (like Terraform) to create a new empty cluster from scratch, install it with ArgoCD, pull all YAML manifests from Git, then restore the database from S3 backups.
RTO: 30–60 minutes. Suitable for non-critical internal applications.

Aligning RTO and RPO Targets #

When designing DR cost budgets, we must determine the following two key metrics together with stakeholders:

Recovery Time Objective (RTO): The maximum system outage duration limit allowed before businesses suffer serious financial losses. “How long can the system be down?”
Recovery Point Objective (RPO): The maximum transaction data loss limit measured by the last write time. “How much transaction data can be lost?”

Here’s a table mapping business target alignment to the technical architecture that must be built:

Business Targets	RTO Targets	RPO Targets	Technical Architecture Strategies
Extreme Critical (Fintech/Core Banking)	< 1 minute	< 1 minute	Active-Active Multi-Region with Multi-Master Database Replication.
Medium Critical (SaaS/E-Commerce)	5 - 15 minutes	< 1 hour	Active-Passive with automatic DNS failover and continuous asynchronous database replication.
Standard Business (Internal Portals)	30 - 60 minutes	< 24 hours	GitOps Cluster Recreation + scheduled Data Restoration from Velero snapshots.