ReplicaSet #

In a Kubernetes cluster serving millions of user requests, we can’t let our applications run unsupervised. If a container dies due to a traffic spike or its physical host server suddenly hangs, the system must immediately detect the failure and stand up a replacement container within milliseconds. The frontline component responsible for keeping our application’s replica count constant is the ReplicaSet.

A ReplicaSet is Kubernetes’ basic supervisor object that ensures the number of Pod replicas we want always runs healthily in the cluster. Although in daily operational practice we rarely create ReplicaSet objects directly, understanding its internal architecture, label selection mechanism, and relationship with Deployment is crucial for preventing Pod placement conflicts and making cluster availability debugging easier.

Why Do We Need a ReplicaSet? The Guardian Loop Concept #

The ReplicaSet philosophy is based on a very simple reconciliation loop that runs endlessly:

[\text{Current State} \stackrel{?}{=} \text{Desired State}]

Every second, the ReplicaSet Controller (running inside kube-controller-manager) counts the number of active Pods in the namespace whose labels match its selector spec.

  • If Current < Desired: The ReplicaSet immediately triggers the creation of new Pods by copying the template structure declared in the manifest.
  • If Current > Desired: (For example, because a node recovered and its old Pods came back to life), the ReplicaSet picks which Pods are excess and sends deletion instructions to rebalance the count.
  • If Current == Desired: No action is taken. The cluster is in a stable state.

This automatic self-healing mechanism cuts application recovery time from minutes (if done manually by a human operator) to just seconds.

The Label Selector Mechanism and Pod Adoption (Ownership) #

One of the unique architectural features distinguishing Kubernetes from traditional orchestration systems is how a ReplicaSet tracks the Pods under its responsibility. A ReplicaSet doesn’t record a static list of Pod names or unique IDs in the database. Instead, it relies on dynamic label matching using a Label Selector.

There are two ways to define the label selector on a ReplicaSet:

  1. matchLabels: Precise, simple key-value pairing.
  2. matchExpressions: Advanced logical expression matching using special operators.

Example manifest using matchExpressions for flexible selection:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: payment-api-rs
  namespace: production
spec:
  replicas: 3
  selector:
    matchExpressions:
      - {key: app, operator: In, values: [payment-api]}
      - {key: environment, operator: NotIn, values: [development, staging]}
      - {key: tier, operator: Exists}
  template:
    metadata:
      labels:
        app: payment-api
        environment: production
        tier: backend
    spec:
      containers:
      - name: main-api
        image: payment-api:v2.0

Explanation of matchExpressions Operators: #

  • In: The Pod label must have the key with one of the values listed in the values array.
  • NotIn: The Pod label must have the key with a value that is not equal to the values in the values array.
  • Exists: The Pod must have a label with that key, regardless of its value (the values property must be left empty when using this operator).
  • DoesNotExist: The Pod must not have that label key.

The Pod Deletion Selection Algorithm (Deletion Priority) #

When the real Pod count exceeds the desired target (e.g. we lower the replica value from 5 to 3), the ReplicaSet Controller doesn’t delete Pods randomly. It runs a special selection algorithm to minimize disruption to cluster traffic:

  1. Pending Pods: Pods still in the Pending phase (not yet running because they’re still pulling images or finding a node) are deleted first.
  2. Readiness: Pods that aren’t ready yet (NotReady / failing the readiness probe) are deleted before healthy (Ready) Pods.
  3. Restart Count: Pods with the most restart history (indicating memory leaks or application instability) are prioritized for deletion.
  4. Creation Time: The youngest Pods are deleted before older Pods to keep long-serving Pods stable.
  5. Node Spread: The Controller tries to keep Pods evenly spread across all cluster nodes. If Node A has 3 Pods and Node B has 1, the Controller deletes a Pod on Node A first.

Dynamic Ownership and Pod Adoption #

  • Adopting Orphan Pods: If we manually create a new Pod (without a ReplicaSet) using kubectl run and attach labels matching the selector, the ReplicaSet immediately detects the Pod’s existence. It adopts it and includes it in the Current State count.
  • OwnerReferences: When a Pod is adopted or created by a ReplicaSet, the API Server writes ownerReferences metadata inside the Pod object, pointing to the parent ReplicaSet name. The Garbage Collector uses this to delete descendant Pods if the ReplicaSet is deleted.

Cascading Deletion Strategies (Garbage Collection) #

When we delete a ReplicaSet object with kubectl delete rs <rs-name>, there are three Garbage Collection policy options:

  1. Background Cascading (Default): The API Server immediately deletes the ReplicaSet object, then in the background the Garbage Collector tracks ownerReferences on Pods and deletes all descendant Pods asynchronously.
  2. Foreground Cascading: The API Server marks the ReplicaSet as “deleting”, deletes all descendant Pods first until clean, then permanently deletes the ReplicaSet object.
  3. Orphan Policy: Deletes only the ReplicaSet object, leaving all descendant Pods alive as orphaned Pods without a supervisor. This option is triggered with the --cascade=orphan flag.

Here’s a flow diagram of how a ReplicaSet processes adoption and Pod count reconciliation:

flowchart TD
    Start["ReplicaSet Reconciliation Loop Tick"] --> GetPods["Find all Pods with the label app=payment-api"]
    GetPods --> CountPods["Count the number of active Pods (Current)"]
    
    CountPods --> Compare{"Is Current == Desired (3)?"}
    
    Compare -- "Yes" --> Sleep["Sleep & Wait for the Next Tick"]
    
    Compare -- "Current < Desired" --> CreatePod["1. Take the Pod spec template\n2. Attach ownerReferences to the ReplicaSet\n3. Send POST /api/v1/pods to create new Pods"]
    
    Compare -- "Current > Desired" --> SelectPod["Pick Pods to delete\n(Prioritizing NotReady / New Pods)"]
    SelectPod --> DeletePod["Send DELETE /api/v1/pods to delete Pods"]
    
    CreatePod --> Sleep
    DeletePod --> Sleep

Layered Abstraction: The Deployment, ReplicaSet, and Pod Relationship #

In modern production environments, we’re strictly forbidden from writing manifests or creating ReplicaSet objects directly. We always use higher-level abstraction objects like Deployment.

Why Must We Use Deployment? #

The ReplicaSet has a big limitation: it can’t do smart application version updates (rolling updates). If we change the image tag from :v1.0 to :v2.0 directly on the ReplicaSet manifest, it won’t do anything to the already-running Pods. We’d be forced to manually delete all Pods one by one so the ReplicaSet creates new Pods with the new image template (causing downtime).

Deployment solves this problem by acting as a manager above the ReplicaSet:

  1. When we deploy a Deployment manifest, the Deployment Controller creates the First ReplicaSet (version 1) and fills it with the version-1 template spec.
  2. The First ReplicaSet creates the version-1 Pods.
  3. When we change the application image in the Deployment manifest to version 2, the Deployment doesn’t edit the First ReplicaSet. Instead, it creates a separate Second ReplicaSet (version 2).
  4. The Deployment gradually scales up the Second ReplicaSet while scaling down the First ReplicaSet to 0.
The Abstraction Hierarchy Structure in Kubernetes:

Deployment: payment-deployment (Manages the release strategy)
   │
   ├── ReplicaSet: payment-deployment-7f4d9 (Active Version - Replicas: 3)
   │     ├── Pod: payment-deployment-7f4d9-abcde
   │     ├── Pod: payment-deployment-7f4d9-fghij
   │     └── Pod: payment-deployment-7f4d9-klmno
   │
   └── ReplicaSet: payment-deployment-1a2b3 (Old Version - Replicas: 0)
         (Kept in etcd for rollback history)

The revisionHistoryLimit property on the Deployment determines how many old ReplicaSet objects (with 0 capacity) are kept in the cluster database. Keeping these old ReplicaSets is crucial because when we roll back via kubectl rollout undo, the Deployment simply reactivates the old ReplicaSet without recreating the structure from scratch, guaranteeing an instant rollback.

Reading Status and Managing ReplicaSets via CLI #

Although managed automatically by Deployment, the skill of reading ReplicaSet status is very valuable during cluster incident investigations.

# Show all ReplicaSets in the active namespace
kubectl get replicasets

The output of the command above looks like this:

NAME                          DESIRED   CURRENT   READY   AGE
payment-deployment-7f4d9b     3         3         3       4d
payment-deployment-1a2b3c     0         0         0       10d
  • DESIRED: The target replica count we requested in the manifest.
  • CURRENT: The actual Pod replica count the ReplicaSet is currently trying to run.
  • READY: The number of healthy Pods that passed the readiness probe and are ready to accept network traffic.

If we see CURRENT increasing but READY staying at 0, we can diagnose the failure by looking at event logs at the ReplicaSet level:

# Check the ReplicaSet's internal event logs
kubectl describe replicaset payment-deployment-7f4d9b

At the bottom of the describe command output, we can detect systemic failures, like namespace quota failures (ResourceQuota Exceeded) or registry authentication rejections when pulling images.

Anti-Patterns in ReplicaSet Management #

Here are fatal mistakes that often disrupt the cluster’s Pod lifecycle monitoring mechanism:

Anti-Pattern 1: Label Selector Conflicts Between ReplicaSets (Label Hijacking) #

Using labels that are too generic or identical for two different controller objects.

ANTI-PATTERN: Writing the app=payment-api Selector for Two Different ReplicaSets/Deployments
// WHAT WE DO:
- Create Deployment A (for payment-api-staging) with the `app: payment-api` label selector.
- Create Deployment B (for payment-api-production) in the same namespace with the `app: payment-api` label selector.

// THE CONSEQUENCES IN PRODUCTION:
- Pod Hijacking: ReplicaSet A and ReplicaSet B compete for the same Pods.
- ReplicaSet A detects 6 active Pods (3 of its own and 3 of B's). Because the desired target is 3,
  ReplicaSet A starts killing 3 Pods at random (possibly killing B's production Pods).
- Conversely, ReplicaSet B detects the Pod count dropping, then creates new Pods. ReplicaSet A kills them again.
- An endless Pod kill-and-create cycle (flapping) occurs, paralyzing the cluster master node's CPU performance.

✓ THE RIGHT SOLUTION:
- Always use unique, specific label combinations to distinguish deployment environments (staging vs production).
- Use industry-standard built-in labels like `app.kubernetes.io/name` combined with an environment label.
- Even safer: use separate namespaces (logical isolation) to separate staging and production.

Anti-Pattern 2: Directly Hot-Patching the Pod Template at the ReplicaSet Level #

Trying to modify the Pod spec directly on a ReplicaSet object managed by a Deployment.

ANTI-PATTERN: Running kubectl edit replicaset <rs-name>
// WHAT WE DO:
- Contact the ReplicaSet binary directly via the command line to change an environment variable.

// THE CONSEQUENCES IN PRODUCTION:
- Wasted Effort: Our change only lasts a few seconds. As soon as the Deployment Controller
  performs its periodic reconciliation, it detects the ReplicaSet spec doesn't match the parent Deployment manifest.
- The Deployment Controller immediately overrides and restores the ReplicaSet spec
  to its original values, wiping out the new configuration we just entered.

✓ THE RIGHT SOLUTION:
- Always make changes at the top level, the `Deployment` manifest.
- Run the edit command on the deployment: `kubectl edit deployment <deploy-name>`
  or update your local YAML manifest file and run `kubectl apply -f deployment.yaml`.

Summary #

  • The Availability Guardian Loop — A ReplicaSet ensures the number of active Pods always exactly matches the desired state number at every second of cluster operation.
  • Label-Based Selection — Pod ownership is determined dynamically via label selectors, not static IDs, enabling flexible Pod adoption.
  • The Danger of Duplicate Pods — Avoid manually creating Pods with labels matching an active ReplicaSet selector to prevent unintentional adoption and force Pod deletion.
  • The Deployment Abstraction — Don’t manage ReplicaSets directly; always use Deployment as the higher-level orchestrator to enable downtime-free rolling updates.
  • Rollback History Storage — Old ReplicaSet objects with 0 capacity are kept by the cluster as history records to speed up instant rollbacks.
  • Incident Identification via Events — Use kubectl describe replicaset to detect cluster quota allocation failures or container image pull errors.
  • Avoid Label Conflicts — Always design unique, specific label selectors for each application to prevent Pod hijacking between controllers.

← Previous: Sidecar Pattern   Next: Deployment →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact