Backup & Restore #
In the dynamic, distributed Kubernetes ecosystem, we’re often lulled by this platform’s self-healing capabilities. We watch dead Pods get immediately replaced by new ones, or failing nodes get handled by moving workloads to healthy nodes. However, we must realize that self-healing is not a substitute for a disaster recovery (DR) and data backup strategy. Data loss from human error, storage backend failures, cyber attacks (ransomware), or regional natural disasters can’t be automatically cured by the Kubernetes control plane.
Backup is our best insurance when operating production systems. This article covers storage backup and restore strategies in Kubernetes in depth, explores the fundamental differences from traditional environments, dissects Kubernetes’ built-in Volume Snapshot architecture, and demonstrates real implementations using Velero and etcd backups.
Why Is Kubernetes Backup Different? #
If you’re used to managing traditional servers or Virtual Machines (VMs), you might be accustomed to one-click backup methods like VM snapshots. In the traditional VM model, a snapshot captures the entire system state: the OS, network configuration, application binaries, memory, and disk data in one unified package.
In Kubernetes, that approach is no longer relevant because of the cluster’s fragmented, distributed architecture. Let’s look at these fundamental differences:
Backup in Traditional Environments (VM):
[OS + App + Configuration + Disk Data]
│
▼ (One Snapshot Unit)
[Single VM Backup Image]
Backup in Kubernetes Environments:
├── Cluster State & Metadata (Deployment, ConfigMap, Secret, RBAC) ──> Stored in Etcd
├── Application State Data (Database, Uploaded Files) ───────────────> Stored in PVCs / Storage Backend
└── Container Images (App Binaries & Libraries) ─────────────────────> Stored in the Container Registry
Therefore, if we want to design a reliable backup system in Kubernetes, we must split our strategy into several layers:
- Cluster State Backup (Etcd): Stores all declarative manifests, resource relationships, security configuration, and the cluster’s current state.
- Volume Data Backup (Persistent Volumes): Stores the real data bits written by our databases or stateful applications to the storage backend.
- Application-Level Backup (Logical Backup): Logically extracts data from inside the application (e.g. SQL database dumps) to guarantee transactional consistency.
Volume Snapshots: The CSI-Based Backup Standard #
Since the Container Storage Interface (CSI) was introduced in Kubernetes, we have a standard way to interact with various third-party storage providers (AWS EBS, GCP Persistent Disk, Azure Disk, Ceph, Rook, etc.). One of CSI’s most important features is the Volume Snapshot API, which lets us take point-in-time snapshots of PersistentVolumeClaims (PVCs) directly through the Kubernetes API.
Volume Snapshot Architecture and Workflow #
The Volume Snapshot manifests in Kubernetes consist of three main resources:
VolumeSnapshotClass: Determines the CSI driver used and the snapshot deletion parameters.VolumeSnapshot: The snapshot creation request by developers (similar to a PVC).VolumeSnapshotContent: The representation of the actual physical snapshot in the storage backend (similar to a PV).
Let’s visualize the Volume Snapshot creation interaction flow with the following diagram:
flowchart TD
User["Developer (VolumeSnapshot)"] -. Creates .-> API["Kubernetes API Server"]
API -. Triggers .-> CSI["CSI Snapshot Controller"]
CSI -. Requests .-> Driver["CSI Driver (AWS/GCP/Ceph)"]
Driver -. Calls API .-> Cloud["Cloud Storage API (EBS Snapshot)"]
Cloud -. Creates physical snapshot .-> Physical["Physical Snapshot (AWS/GCP)"]1. Creating a VolumeSnapshotClass #
Before we can create snapshots, the cluster administrator must define a VolumeSnapshotClass. This resource determines which driver is responsible for taking snapshots and how the snapshot lifecycle is managed.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-aws-ebs-snapshotclass
driver: ebs.csi.aws.com
deletionPolicy: Retain
parameters:
tagSpecification_1: "key=Environment,value=Production"
tagSpecification_2: "key=BackupType,value=CSI-Snapshot"
driver: Must match the active CSI driver in our cluster (e.g.ebs.csi.aws.comfor AWS EBS).deletionPolicy: Determines whether the physical cloud snapshot gets deleted (Delete) or retained (Retain) when theVolumeSnapshotobject in Kubernetes is deleted. We highly recommend choosingRetainfor production environments to avoid accidental data loss.
2. Submitting a VolumeSnapshot Creation Request #
Once the VolumeSnapshotClass is available, developers can snapshot a specific PVC by creating a VolumeSnapshot manifest.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-data-snapshot-20260617
namespace: database
spec:
volumeSnapshotClassName: csi-aws-ebs-snapshotclass
source:
persistentVolumeClaimName: postgres-pvc-postgres-0
We can monitor the snapshot creation process with:
kubectl get volumesnapshot -n database
The output we’ll see:
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE AGE
postgres-data-snapshot-20260617 true postgres-pvc-postgres-0 snapcontent-123456... 50Gi 2m
Make sure the READYTOUSE column is true, indicating the snapshot was successfully created physically in the storage backend and is ready for restore.
3. Restoring Data from a Volume Snapshot #
When we want to restore our data from a created snapshot, we don’t overwrite the old PVC. Instead, we create a new PVC specifying a dataSource pointing to our VolumeSnapshot object.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-pvc-restored
namespace: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: premium-rwo
resources:
requests:
storage: 50Gi
dataSource:
name: postgres-data-snapshot-20260617
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
When we apply this manifest, the CSI driver detects the dataSource, creates a new volume on the cloud provider based on the stored physical snapshot, then binds that volume to this new PVC object. We can then mount the postgres-pvc-restored PVC to our new database Pod.
Logical Backups: Application-Level Consistency #
Although CSI-based Volume Snapshots are very fast and efficient, we must understand their limitations. CSI snapshots work at the block storage level (crash-consistent). That means snapshots only guarantee the filesystem structure is safe, but don’t guarantee consistency for application data sitting in memory (buffer pool or caching layer).
If we snapshot a PostgreSQL or MySQL database actively receiving write transactions without locking or flushing, there’s a high chance the transaction data in the snapshot becomes inconsistent (corrupted).
To solve this problem, we must apply Logical Backups (Application-Level Backups) using the database’s built-in utilities like pg_dump, mysqldump, or mongodump.
Implementing a Postgres Backup CronJob to Object Storage (S3) #
Let’s create a reliable CronJob manifest for running regular logical PostgreSQL database backups every day at 02:00 AM. This backup gets compressed and shipped to an external AWS S3 bucket immediately, leaving no junk files on the node.
First, we prepare a Secret containing our AWS S3 and database credentials:
apiVersion: v1
kind: Secret
metadata:
name: postgres-backup-secret
namespace: database
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: "AKIAEXAMPLEACCESSKEY123"
AWS_SECRET_ACCESS_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
POSTGRES_PASSWORD: "SuperSecurePostgresPassword123"
Next, we define our CronJob manifest:
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgres-logical-backup
namespace: database
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: postgres-backup-agent
image: amazon/aws-cli:2.15.15
command:
- /bin/sh
- -c
- |
set -e
echo "Starting logical database backup..."
# Install the postgresql client at runtime if the aws-cli image doesn't have it
yum install -y postgresql15
BACKUP_DATE=$(date +%Y-%m-%d-%H%M%S)
FILENAME="postgres-backup-${BACKUP_DATE}.sql.gz"
echo "Dumping database 'app_production'..."
PGPASSWORD="${POSTGRES_PASSWORD}" pg_dump -h postgres-service.database.svc.cluster.local -U postgres -d app_production | gzip > "/tmp/${FILENAME}"
echo "Uploading backup to AWS S3..."
aws s3 cp "/tmp/${FILENAME}" "s3://production-k8s-database-backups/postgres/${FILENAME}"
echo "Backup and upload process completed successfully!"
rm -f "/tmp/${FILENAME}"
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-backup-secret
key: POSTGRES_PASSWORD
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: postgres-backup-secret
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: postgres-backup-secret
key: AWS_SECRET_ACCESS_KEY
- name: AWS_DEFAULT_REGION
value: "ap-southeast-1"
concurrencyPolicy: Forbid: Guarantees that if today’s backup runs slowly (because data size ballooned), Kubernetes won’t run a new parallel backup job the next day that could disrupt database performance.set -e: Ensures that if any command line fails (e.g. database connection failure or S3 upload failure), the container immediately exits with an error status so the scheduler knows the backup failed and triggers an alert.
Velero: The Comprehensive Disaster Recovery Solution #
If we want to secure our entire cluster holistically (not just disk data, but also all API Server configuration manifests), we need a special tool. Velero (formerly known as Heptio Ark) is the most popular open-source industry standard for backing up and restoring entire Kubernetes clusters.
Velero works by capturing the state of Kubernetes API objects and storing them as tarballs in Object Storage, while integrating with the CSI Volume Snapshot API to back up PVC data.
Let’s look at Velero’s workflow during backup and restore:
flowchart TD
Velero["Velero CLI / CRD"] -. Triggers Backup .-> Controller["Velero Controller"]
Controller -. Queries Resources .-> API["Kubernetes API Server"]
Controller -. Backs Up Manifests .-> S3["Object Storage (S3/GCS)"]
Controller -. Snapshots Volumes .-> CSI["CSI Volume Snapshot"]
CSI -. Creates Snapshot .-> Storage["Cloud / Persistent Storage"]1. Preparing the Velero Installation #
To install Velero in our cluster, we must prepare an external Object Storage bucket (e.g. S3) and the appropriate access credentials. Save our AWS credentials in a local file named credentials-velero:
[default]
aws_access_key_id = AKIAEXAMPLEACCESSKEY123
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Then, run the following installation command using the Velero CLI:
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.9.0 \
--bucket production-k8s-velero-backups \
--backup-location-config region=ap-southeast-1 \
--snapshot-location-config region=ap-southeast-1 \
--secret-file ./credentials-velero \
--use-volume-snapshots=true
The command above creates the velero namespace, installs the required Custom Resource Definitions (CRDs), deploys the Velero controller, and configures the integration with our AWS S3 bucket.
2. Performing Backups with Velero #
Once Velero is installed, we can back up an entire namespace (including Deployment, Service, ConfigMap, Secret, and PVC manifests) with:
velero backup create production-ns-backup-20260617 \
--include-namespaces production \
--snapshot-volumes
If we want to create an automatic backup schedule (e.g. daily at 03:00 AM), we can create a Velero Schedule resource:
velero schedule create daily-production-backup \
--schedule="0 3 * * *" \
--include-namespaces production \
--ttl 168h0m0s
--ttl 168h0m0s: Sets the backup Time To Live (retention) for 7 days (168 hours). Velero automatically deletes manifest tarballs and cloud disk snapshots older than that limit to save storage costs.
3. Performing Restores with Velero #
If a total disaster occurs and our cluster is severely damaged, we can create a new empty cluster, install Velero with the same bucket configuration, then pull the existing backup:
# List the available backups in the bucket
velero backup get
Once we find the backup we want, run the restore command:
velero restore create --from-backup production-ns-backup-20260617 \
--include-namespaces production \
--restore-volumes=true
Velero reads the tarball files from the bucket, reconstructs all objects in the Kubernetes API Server in the correct order (e.g. Namespaces first, then Secrets and PVCs, then Deployments), and restores our data volumes from the CSI snapshots.
Etcd Backup: The Security Heart of Self-Managed Clusters #
If we operate a self-managed Kubernetes cluster on bare-metal VMs using utilities like kubeadm or kops, we have full responsibility for our cluster’s control plane data stored in etcd.
Etcd is the distributed key-value database holding the single source of truth for Kubernetes cluster configuration and state. Losing the etcd database without a backup means permanently losing our entire cluster.
[!NOTE] For those of us using managed services (like AWS EKS, GCP GKE, or Azure AKS), the control plane and etcd database are fully managed by the cloud provider. Therefore, we don’t need to think about etcd backups and can focus our strategy on PVC data and application manifest backups.
1. Creating an Etcd Snapshot Manually #
We must execute the etcdctl utility directly on the control plane node where etcd runs. Because etcd communicates using TLS, we must include our cluster’s authentication certificates.
Run the following command on the control plane node:
ETCDCTL_API=3 etcdctl snapshot save /var/lib/db-backup/etcd-snapshot-20260617.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
After the snapshot is created, we must verify the database file’s integrity:
ETCDCTL_API=3 etcdctl snapshot status /var/lib/db-backup/etcd-snapshot-20260617.db --write-out=table
A valid verification output shows the database hash, file size, and last transaction index:
+----------+----------+------------------+------------------+
| HASH | REVISION | TOTAL KEYS (EST) | TOTAL SIZE (EST) |
+----------+----------+------------------+------------------+
| c8e9e1a7 | 24950381 | 5204 | 4.8MB |
+----------+----------+------------------+------------------+
Don’t forget to immediately ship this .db snapshot file to an external server or Object Storage using a separate cron job, so the backup data doesn’t get destroyed if that control plane node explodes or suffers hardware damage.
2. Restoring the Cluster Using an Etcd Snapshot #
If our main etcd database is corrupted, we can return the cluster state to the point when the snapshot was taken. This process requires stopping the Kubernetes controller services first so no new transactions happen while data is being overwritten.
# 1. Stop kube-apiserver and etcd on the control plane node (move the static manifests)
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
# 2. Delete or back up the current corrupted etcd data directory
mv /var/lib/etcd /var/lib/etcd-corrupted
# 3. Run the snapshot restore to a new directory
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/db-backup/etcd-snapshot-20260617.db \
--data-dir=/var/lib/etcd \
--initial-cluster=control-plane-node=https://127.0.0.1:2380 \
--initial-cluster-token=etcd-cluster-1 \
--initial-advertise-peer-urls=https://127.0.0.1:2380 \
--name=control-plane-node
# 4. Restore the etcd and kube-apiserver static manifests so the control plane comes back up
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
After the kubelet reloads the control plane objects, we can check cluster health again with kubectl get nodes.
Anti-Patterns vs Solutions in Backup & Restore #
Let’s study some fatal mistakes (anti-patterns) often made when designing data backup strategies in Kubernetes, along with example code to fix them.
Anti-Pattern 1: Taking DB Snapshots Without Fsync Lock/Logical Dumps #
We take a volume snapshot of an actively running PostgreSQL or MySQL database processing thousands of write transactions per second without temporarily pausing the disk write process. This produces a dirty disk snapshot vulnerable to read process failures when restored.
Wrong Manifest Code (Direct Snapshot Without Data Protection) #
# DON'T DO THIS ON A BUSY DATABASE
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-dirty-snapshot
namespace: database
spec:
volumeSnapshotClassName: csi-aws-ebs-snapshotclass
source:
persistentVolumeClaimName: postgres-pvc-0 # Actively running database PVC snapshotted directly
Solution Code (Using Pre-Snapshot Hooks via Velero) #
We’re advised to use Velero Container Hooks to tell the database to flush all in-memory data to disk and lock write transactions just before the snapshot is taken, then release the lock after the snapshot finishes.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: database
spec:
serviceName: postgres-service
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
annotations:
# Pre-backup hook: Lock database writes
pre.backup.velero.io/command: '["/bin/sh", "-c", "psql -U postgres -c \"SELECT pg_backup_start(''velero-backup'');\""]'
pre.backup.velero.io/container: "postgres-db"
# Post-backup hook: Release the write lock after the snapshot is taken
post.backup.velero.io/command: '["/bin/sh", "-c", "psql -U postgres -c \"SELECT pg_backup_stop();\""]'
post.backup.velero.io/container: "postgres-db"
spec:
containers:
- name: postgres-db
image: postgres:15
# ... rest of container spec ...
Anti-Pattern 2: Storing Backups on Local Worker Node Disks (hostPath)
#
We create a database backup CronJob and store it in a local node directory using the hostPath volume type. If that worker node dies from hardware failure, our backup data dies with the node.
Wrong Manifest Code (Relying on a Node’s Local Disk) #
# DON'T DO THIS: Backups stored on a physical node
apiVersion: batch/v1
kind: CronJob
metadata:
name: insecure-local-backup
namespace: database
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: mysql-backup
image: mysql:8.0
command: ["/bin/sh", "-c", "mysqldump -h mysql-service -u root -p$MYSQL_ROOT_PASSWORD appdb > /backup/appdb.sql"]
env:
- name: MYSQL_ROOT_PASSWORD
value: "password"
volumeMounts:
- name: backup-dir
mountPath: /backup
volumes:
- name: backup-dir
hostPath:
path: /var/lib/mysql-backups # Data is bound to one physical node only!
type: DirectoryOrCreate
Solution Code (Streaming Directly to External Cloud Storage) #
We must stream the backup output directly to external Object Storage (like AWS S3, Google Cloud Storage, or Azure Blob) over a secure network without permanently storing it on the worker node’s local disk.
# SOLUTION: Using AWS CLI S3 integration via pipe stream
apiVersion: batch/v1
kind: CronJob
metadata:
name: secure-cloud-backup
namespace: database
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: mysql-backup
image: mysql:8.0
command:
- /bin/sh
- -c
- |
# Stream the database dump directly to AWS S3 via CLI pipe
mysqldump -h mysql-service -u root -p"${MYSQL_ROOT_PASSWORD}" appdb | gzip | aws s3 cp - s3://my-cloud-backup-bucket/mysql/backup-$(date +%Y%m%d).sql.gz
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: database-password
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-credentials
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-credentials
key: secret-access-key
Strategy Recommendations & DR Drills #
Having data backups doesn’t guarantee successful disaster recovery if we never periodically test the restore process. As a guide for platform teams, let’s follow these principles:
1. Apply the 3-2-1 Backup Rule #
- 3 Data Copies: We should have at least three data copies: 1 active copy in the production cluster, 1 fast CSI volume snapshot backup in the local cloud backend, and 1 logical backup (e.g. SQL dump files).
- 2 Different Media: Store backup data on at least two separate storage media types (e.g. block storage volume snapshots and object storage buckets).
- 1 Offsite Location: Make sure at least one backup copy is stored outside the main geographic region of our production cluster (e.g. if the cluster runs in AWS region
ap-southeast-1Singapore, store offsite backups in regionap-southeast-3Jakarta) to anticipate total regional data center failures by the cloud provider.
2. Perform Periodic Disaster Recovery (DR) Drills #
Too often we only realize our backup files are corrupted or incomplete during an emergency. To prevent this disaster:
- Schedule regular data recovery drills (DR Drills), at least once a month.
- Use a non-production cluster (staging or testing) to simulate total failure, and perform data restore using Velero or CSI snapshot manifests.
- Measure the RTO (Recovery Time Objective - how long until the cluster is back online) and RPO (Recovery Point Objective - how much recent transaction data is lost due to backup time gaps) and align them with our company’s business needs.
Summary #
- Kubernetes backups are fragmented: We must split the backup strategy into three main pillars: Cluster State (Etcd), Volume Data (Persistent Volumes), and Application Logic (Database dumps).
- CSI Volume Snapshots: Help us take efficient point-in-time disk backups at the infrastructure level. We can restore them by defining new PVC objects using
dataSource.- Logical Dumps for Databases: Highly recommended for active databases to avoid data inconsistency risks (corrupted state) common in crash-consistent snapshot models.
- Velero as the Main Solution: Use Velero to capture the cluster state comprehensively and automate the Kubernetes manifest backup/restore cycle along with volume data to external Object Storage.
- Protect Etcd on Self-Managed Clusters: If managing the control plane ourselves, scheduled etcd backups with
etcdctl snapshot saveare an absolute obligation.- Run Routine Simulations: Data backups whose restore process was never tested have zero percent recovery value. Always run DR drills at least once a month.