Etcd & Cluster Consistency #

In the distributed Kubernetes system, data consistency is everything. If we deploy an application, restart a node, or change network rules, all that information must be stored somewhere that is not only safe from physical failure but also guarantees absolute data accuracy on every node. The single database carrying this heavy responsibility is etcd.

If the API Server is the cluster’s front door, etcd is the memory or brain of the entire Kubernetes cluster. All configuration state, metadata, and the real status of every resource — from tiny Pods to sensitive Secrets — are stored in etcd. Understanding how etcd works, its consensus math, and how to maintain its health is a mandatory prerequisite for DevOps teams and system administrators to prevent permanent loss of production cluster data.

The Raft Consensus Algorithm and Quorum Math #

etcd isn’t a traditional database running on a single server. To avoid a single point of failure (SPOF), etcd runs as a distributed cluster of several nodes (usually 3 or 5 instances). The main challenge of a distributed database is ensuring every node always has exactly the same data records at every second.

To solve this problem, etcd uses the Raft Consensus Algorithm. The Raft algorithm divides runtime into election terms and assigns three roles to nodes: Leader, Follower, and Candidate.

flowchart TD
    subgraph RaftConsensus["Raft Log Replication Consensus Process"]
        direction TB
        Client["API Server (Client)"] -->|1. Write Data: Pod A = Running| Leader["etcd Leader (Node 1)"]
        
        Leader -->|2. Send Log AppendEntries| Follower1["etcd Follower (Node 2)"]
        Leader -->|2. Send Log AppendEntries| Follower2["etcd Follower (Node 3)"]
        
        Follower1 -->|3. Send ACK| Leader
        Follower2 -->|3. Send ACK| Leader
        
        NoteOverLeader{"Is the ACK Received by the Majority?"}
        Leader --> NoteOverLeader
        
        NoteOverLeader -- "Yes (Quorum Met)" --> Commit["4. Commit Log to Disk & etcd DB"]
        Commit --> RespondClient["5. Send HTTP 200 OK to the API Server"]
        
        Commit -.->|6. Propagate Commit Event| Follower1
        Commit -.->|6. Propagate Commit Event| Follower2
    end

    style Leader stroke:#2ecc71,stroke-width:2px
    style Follower1 stroke:#3498db,stroke-width:2px
    style Follower2 stroke:#3498db,stroke-width:2px
    style Commit stroke:#e74c3c,stroke-width:2px

Quorum Math: Why Must the Node Count Be Odd? #

In the Raft algorithm, a data write transaction is only declared committed when the transaction log has been successfully written to physical disk by a majority of the nodes in the etcd cluster. This majority is mathematically known as the Quorum.

The etcd quorum calculation formula is:

[\text{Quorum} = \text{floor}\left(\frac{N}{2}\right) + 1]

Where (N) represents the total number of etcd nodes in the cluster.

Based on the formula above, let’s analyze the comparison of node counts against the cluster’s fault tolerance limits:

Avoiding Split-Brain #

Using an odd node count guarantees that if a network disruption splits the cluster into two physically separate parts (network partition), only one region can maintain quorum.

For example, on a 5-node cluster split into region A (3 nodes) and region B (2 nodes):

• Region A has 3 nodes, meeting the minimum quorum ((>= 3)). This region can keep processing read and write transactions normally.
• Region B only has 2 nodes, below the minimum quorum. This region automatically rejects write transactions to prevent divergent data branching (Split-Brain). When the network recovers, nodes in Region B sync their data with Region A.

The MVCC Storage Model and Defragmentation #

etcd adopts the MVCC (Multi-Version Concurrency Control) storage model. Unlike traditional databases that directly overwrite old data during updates (in-place updates), etcd never deletes old data directly.

Every time data is updated, etcd creates a new copy version with a globally increasing Revision number. This revision number acts as a logical timestamp.

• Reading past data: Because it keeps version history, etcd lets the API Server read the cluster object state at a specific revision from the past.
• Efficient Watch Mechanism: Revisions act as tracking anchors. If the network connection between the API Server and etcd drops for a few seconds, the API Server can reconnect and request: “Send me all change events since revision number 10450”.

Internally, etcd maps data in two layers:

1. B-tree in Memory (RAM): Stores an index of key names and revision number history.
2. bbolt DB on Disk (Storage): A simple transactional key-value database storing revision numbers as the real keys and the Kubernetes JSON object contents as the values.

The Disk Space Fragmentation Problem #

Because etcd keeps storing new revision versions, the physical disk capacity of the database grows over time. To prevent running out of capacity, etcd performs periodic Compaction (Kubernetes typically triggers this automatically every 5 minutes). Compaction deletes all historical revision records before a certain revision boundary.

However, the Compaction process doesn’t return that freed disk space to the OS. The freed space stays locked by etcd and is marked as unused internal space. This condition is called Fragmentation. If fragmentation is left unhandled, etcd hits the maximum database quota (by default ranging from 2 GB to 8 GB) and automatically locks the cluster into Read-Only status (database space exceeded).

To physically free disk space and return it to the host OS, we must run the Defragmentation process.

# Compact/compress manually up to revision 100000
ETCDCTL_API=3 etcdctl compact 100000 \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Run defragmentation to free physical disk space (do it one node at a time)
ETCDCTL_API=3 etcdctl defrag \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcd Backup and Restore Strategy for Disaster Recovery #

Many DevOps teams mistakenly assume that backing up application YAML manifests (e.g. using GitOps) is enough. Git manifests don’t store the cluster’s dynamic operational state such as Service IP allocations, cloud storage volume status, certificate metadata, operator status, and in-flight deployment history. The only way to fully restore a cluster after an infrastructure disaster is restoring the etcd data snapshot.

Step 1: Taking a Valid etcd Snapshot #

To take a consistent etcd snapshot, we must use the etcdctl tool with the cluster’s TLS security certificates:

# Backup the etcd snapshot
ETCDCTL_API=3 etcdctl snapshot save /var/lib/backup/etcd-snapshot-prod.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

After the snapshot is saved, make sure to verify the snapshot file’s integrity with the status command:

# Verify the database snapshot integrity
ETCDCTL_API=3 etcdctl snapshot status /var/lib/backup/etcd-snapshot-prod.db --write-out=table

Step 2: The Snapshot Restore Process #

[!CAUTION] The etcd restore process is destructive. This action deletes all current cluster data and rewinds the cluster state to when the snapshot was taken. During the restore, all Kubernetes controller processes must be stopped temporarily.

To restore a multi-node cluster from a snapshot, we must run the restore command on each etcd node independently before starting the API Server processes:

# Run the data restore on Node-1
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/backup/etcd-snapshot-prod.db \
  --name=etcd-node-1 \
  --data-dir=/var/lib/etcd-restored \
  --initial-cluster=etcd-node-1=https://10.240.0.10:2380,etcd-node-2=https://10.240.0.11:2380,etcd-node-3=https://10.240.0.12:2380 \
  --initial-cluster-token=etcd-token-token-baru \
  --initial-advertise-peer-urls=https://10.240.0.10:2380

After running the restore command on all nodes, we need to:

1. Change the etcd data directory path in the etcd host manifest configuration (usually at /etc/kubernetes/manifests/etcd.yaml) to point to /var/lib/etcd-restored.
2. Restart the Kubelet service to reload the new configuration.
3. Check cluster health with kubectl get nodes.

Disk Write Speed Requirements (fsync) and SSD #

etcd is very sensitive to disk I/O latency. Every time data is written to etcd, the Raft algorithm requires that log data be permanently stored to physical disk using the fsync system operation before the transaction is confirmed to the cluster.

The fsync operation forces the Linux OS to ignore its memory cache and write data directly to physical disk. On traditional mechanical disks (HDD) or network storage media (network storage like non-SSD AWS EBS), this operation takes a long time.

Visualizing the Impact of High fsync Latency on etcd:

flowchart TD
    A["High Disk Latency (HDD / Slow Cloud Storage)"] --> B["fsync operations take > 10ms"]
    B --> C["Raft heartbeats between etcd nodes arrive late"]
    C --> D["Followers think the Leader is dead (no heartbeat)"]
    D --> E["Triggers repeated new Leader elections (Leader Election Drop)"]
    E --> F["The cluster experiences total instability (partial split-brain & write timeouts)"]

To guarantee production cluster stability, etcd requires:

• fsync latency at the 99th percentile (p99) must stay below 10 milliseconds (ideally below 1 millisecond).
• We must use solid-state drives (SSD) with capable IOPS. Placing the etcd data directory on mechanical hard disks (HDD) or shared network filesystems (like NFS) is strictly forbidden.

Anti-Patterns in etcd Management #

Here are two fatal mistakes system administrator teams often make when managing etcd in production:

Anti-Pattern 1: Using HDD or Shared Storage for the etcd Data Directory #

A fatal mistake in choosing the cluster database’s physical storage medium.

ANTI-PATTERN: Placing the etcd Data Directory on Slow Network Storage
// WHAT WE DO:
- Run a self-managed etcd cluster in the AWS cloud environment.
- Store the `/var/lib/etcd` data directory on a `gp2` EBS storage volume
  with low IOPS without dedicated IOPS allocation (PIOPS).
- Or place the data directory on a shared network filesystem like NFS.

// THE CONSEQUENCES IN PRODUCTION:
- Repeated Leadership Loss: When cluster load spikes (e.g. deploying 100 Pods at once),
  fsync latency soars past >100ms. etcd nodes lose Raft heartbeat communication.
- API Server Failure: The API Server returns `503 Service Unavailable` errors in succession
  to clients because the etcd database connection drops due to an unfinished leader election.
- Data Corruption: If the network partition flaps rapidly, data can become inconsistent.

✓ THE RIGHT SOLUTION:
- Always use local SSDs (like NVMe) physically attached to the server for the etcd data directory.
- If using a cloud provider (AWS), use `gp3` volumes with a minimum 3000 IOPS allocation,
  or use `io2` (Provisioned IOPS) to guarantee fsync latency stays below 1ms.
- Use the `fio` utility to benchmark disk performance before installing etcd on the server:
  fio --name=write_latency --filename=test.fio --ioengine=psync --rw=write --fsync=1 --bufsize=4k --size=10m

Anti-Pattern 2: Ignoring Database Quota Settings and Log Cleanup (No Compaction) #

Letting the etcd database grow without limits until it unilaterally locks the system.

ANTI-PATTERN: Running etcd Without Auto-Compaction
// WHAT WE DO:
- Configure the etcd cluster without the `--auto-compaction-retention` argument.
- Perform deployment, scaling, and deletion of thousands of Pods every day in a busy CI/CD environment.

// THE CONSEQUENCES IN PRODUCTION:
- Quota Space Exhaustion: Revision modification history keeps piling up in the bbolt DB file.
  After the database hits the default limit (2 GB), etcd detects the `NOSPACE` alarm and locks the database.
- Total Cluster Paralysis: The Kubernetes cluster suddenly turns Read-Only.
  Admins can't run `kubectl delete`, `kubectl edit`, or even scale applications down to relieve the load.

✓ THE RIGHT SOLUTION:
- Always enable the auto-compaction feature in the etcd startup configuration with a specific retention duration, e.g. 1 hour:
  --auto-compaction-retention=1
- Create a daily cron job script or use Prometheus monitoring alerts to detect etcd storage capacity.
- If the cluster is already locked by the NOSPACE alarm, take these rescue steps:
  1. Get the list of active alarms:
     `etcdctl alarm list`
  2. Get the latest revision:
     `etcdctl endpoint status --write-out=table`
  3. Run manual compaction up to the latest revision number.
  4. Run disk defragmentation sequentially on each node.
  5. Clear the alarm locking the cluster:
     `etcdctl alarm disarm`

Summary #

• The Cluster’s Source of Truth — etcd stores all Kubernetes configuration state and metadata; losing etcd without a backup means permanently losing the entire cluster state.
• Strong Raft Consensus — etcd uses the Raft algorithm to guarantee strong consistency, where data is only considered committed after being successfully written by a majority of nodes.
• The Odd Quorum Rule — Always use an odd node count (3 or 5) to optimize fault tolerance and avoid data split-brain risk during network partitions.
• The MVCC Storage System — etcd doesn’t overwrite old data but creates new revision copies, letting the API Server reliably track the history of cluster object state changes.
• Disk Space Management — Run Compaction regularly to discard old revision history, followed by Defragmentation to return freed disk space to the OS.
• Snapshots for Disasters — Backing up YAML manifests isn’t enough; we must schedule periodic binary etcd snapshot captures and test the restore process regularly.
• SSD Is an Absolute Requirement — Disk fsync operation latency must stay below 10ms (ideally <1ms); using HDDs or slow network storage will break the Raft consensus communication.

← Previous: Controller Manager   Next: Pod Anatomy →