Controller Manager #

In a Kubernetes cluster, we often hear the term “declarative configuration”. As users, we only need to state the cluster’s desired end state through YAML manifests — for example, asking for three replicas of an application container running stably. However, the system doesn’t magically reach that state by itself. Behind the scenes, there’s an execution engine component working day and night to match the real condition on the ground (current state) with our desires. That vital component is kube-controller-manager (the Controller Manager).

As the main manager of various controllers, the Controller Manager packages dozens of independent control loops into a single binary process. Understanding the Controller Manager’s internal architecture helps us detect why configuration changes aren’t applied immediately, design highly fault-tolerant systems, and grasp the foundations of writing Custom Controllers (the Operator Pattern) to automate complex infrastructure.

The Control Loop Concept and the Reconciliation Cycle #

The heart of all Kubernetes automation lies in a simple yet very powerful pattern called the reconciliation loop. This pattern acts like the AC thermostat in our home: the thermostat observes the current room temperature, compares it with the target temperature we set, then turns the cooling compressor on or off to reach that target continuously.

In Kubernetes, the reconciliation loop formula can be summarized in three repeating steps:

1. Observe: Read the latest state of cluster objects through a local cache connected to the API Server.
2. Compare: Analyze the difference between the current real state and the desired state listed in the resource spec.
3. Act: Make API calls to create, update, or delete cluster resources so the real state approaches the desired state.

Here’s a visual flow of the reconciliation loop running endlessly inside every controller:

flowchart TD
    Start["Start the Reconciliation Cycle"] --> Observe["1. Observe\nRead the real cluster state via the API Server cache"]
    Observe --> Compare["2. Compare\nCheck if Current State == Desired State"]
    Compare --> Decision{Is there a difference?}
    
    Decision -- "No (Stable)" --> Wait["Wait for New Event / Timer Tick"]
    Decision -- "Yes (Inconsistent)" --> Act["3. Act (Execute Action)\nSend state change instructions to the API Server"]
    
    Act --> UpdateCache["Update Local Cache"]
    UpdateCache --> Wait
    Wait --> Observe

Idempotency and Failure Resilience Characteristics #

The reconciliation loop pattern is designed to be idempotent. That means no matter how many times the loop runs with the same input data, the final system state stays consistent without triggering damaging side effects. This property is crucial in distributed systems:

• Crash Resilience: If the kube-controller-manager binary suddenly crashes and dies mid-way, the controller doesn’t need to remember its last step in detail. When the binary restarts, it just compares the latest state from the API Server and takes reconciliation action from scratch.
• Event-Driven & Level-Driven: Controllers don’t just react to momentary change signals (edges); they hold firmly to level states (levels). This guarantees that even if change notifications are lost or missed due to connection issues, the next reconciliation automatically detects the difference when re-reading the cluster state.

Exploring the Core Cluster Controllers #

Although packaged in one binary process (kube-controller-manager), this component runs dozens of logically isolated internal controllers. Each controller is responsible for the lifecycle of a specific Kubernetes object. Here’s an in-depth exploration of the most crucial controllers in the cluster:

1. Node Lifecycle Controller #

The Node Controller is fully responsible for monitoring the physical and virtual health of Worker Nodes registered in the cluster.

• Heartbeat Mechanism: The kubelet on every node periodically sends health reports to the API Server. The Node Controller monitors when those reports arrive.
• Grace Period: If a node doesn’t send heartbeat reports for longer than the tolerance window (by default node-monitor-grace-period is 40 seconds), the Node Controller updates that node’s status to Unknown or NotReady.
• Eviction: If the node stays unhealthy past the eviction timeout (default pod-eviction-timeout is 5 minutes), the Node Controller adds a special taint (node.kubernetes.io/unreachable or node.kubernetes.io/not-ready) to the Node object. This triggers automatic eviction of Pods without a matching toleration, then reschedules them to another healthy Worker Node.

2. ReplicaSet Controller #

The ReplicaSet Controller ensures the number of running Pod replicas in the cluster always exactly matches the number defined in the .spec.replicas property.

• OwnerReferences: To identify which Pods are under its supervision, the ReplicaSet Controller uses a Label Selector. Pods created by a ReplicaSet carry ownerReferences metadata pointing to the parent ReplicaSet.
• Reconciliation Flow: If the number of active Pods with matching labels is less than the desired state (e.g. because a Pod was manually deleted), the ReplicaSet Controller sends a POST request to the API Server to create a new Pod. Conversely, if Pods exceed the desired state, the controller deletes the extra Pods in an orderly fashion (prioritizing Pods that aren’t ready yet or were just created).

3. Deployment Controller #

The Deployment Controller sits at a higher abstraction level. It doesn’t manage Pods directly; instead, it manages ReplicaSet objects to facilitate downtime-free application updates.

• Rolling Update: When we update the application image version in a Deployment manifest, the Deployment Controller creates a new ReplicaSet object (version 2). It then gradually raises the replica capacity on the new ReplicaSet while lowering the replica capacity on the old ReplicaSet (version 1) in parallel.
• Transition Latency Control: Two important parameters managed by this controller are maxSurge (the maximum number of Pods allowed above the target during the update) and maxUnavailable (the maximum number of Pods allowed to be unavailable during the update).
• Rollback History: The Deployment Controller keeps a history of old ReplicaSet objects (set by revisionHistoryLimit). If we issue a rollback command, the controller simply reverses the replica capacity synchronization direction back to the previous ReplicaSet version.

4. EndpointSlice Controller #

In the past, Kubernetes used a traditional Endpoint Controller that stored all Pod IPs in a single Endpoint object per Service. On large-scale clusters with thousands of Pods, modifying one Pod IP forced the entire IP list to be rewritten to the API Server, causing extremely high network and etcd load.

• EndpointSlice Abstraction: To solve that scalability problem, the EndpointSlice Controller splits the Service’s target IP list into several small objects called EndpointSlice (each holding a maximum of 100 endpoints by default).
• Realtime Synchronization: The EndpointSlice Controller watches Pod lifecycle changes and readiness (readiness probe). When a Pod is declared healthy by the Kubelet, the controller adds the Pod’s IP to the appropriate EndpointSlice, so kube-proxy on every node can immediately allow network traffic into that new container.

5. ServiceAccount Controller & Garbage Collector #

• ServiceAccount Controller: Automates the creation of default security tokens for every namespace. This controller watches for new namespace creation and immediately provisions the default ServiceAccount along with the authentication token Secret needed so containers in that namespace can interact with the API Server when permitted.
• Garbage Collector: Responsible for deleting orphaned objects whose parents have been deleted. This mechanism uses ownerReferences metadata with Cascade cleanup rules (e.g. if a Deployment object is deleted, the Garbage Collector tracks and cleanly deletes all associated ReplicaSet and Pod child objects).

Leader Election in High Availability (HA) #

In production environments demanding high availability, we can’t run just one Control Plane instance. We must run at least three Control Plane instances in parallel. However, this poses a serious challenge for the Controller Manager: because controllers actively write state to the API Server, if two Controller Manager instances actively write the same instructions simultaneously, data conflicts (split-brain) trigger cluster inconsistency.

To prevent this problem, Kubernetes adopts the Leader Election pattern based on the Active-Passive model (one active instance, others standby).

sequenceDiagram
    participant InstanceA as Controller Manager A (Standby)
    participant InstanceB as Controller Manager B (Active Leader)
    participant APIServer as API Server (Lease API)
    
    Note over InstanceB: Renews the Lease lock every 2 seconds
    InstanceB->>APIServer: UPDATE leases/kube-controller-manager (renewTime = Now)
    APIServer-->>InstanceB: 200 OK
    
    Note over InstanceA: Watches the Lease status periodically
    InstanceA->>APIServer: GET leases/kube-controller-manager
    APIServer-->>InstanceA: Lease Data (Leader: Instance B, duration: 15s)
    
    Note over InstanceB: Instance B crashes completely!
    
    Note over InstanceA: Lease time expires (> 15 seconds)
    InstanceA->>APIServer: GET leases/kube-controller-manager
    APIServer-->>InstanceA: Lease Data (Expired Lease)
    
    InstanceA->>APIServer: UPDATE leases/kube-controller-manager (Leader: Instance A)
    APIServer-->>InstanceA: 200 OK (Instance A Becomes the New Leader)
    Note over InstanceA: Starts running its internal reconciliation loops

How the Lease API Works #

Kubernetes uses the Lease object under the coordination.k8s.io API group as a distributed lock. Here’s an example manifest representing this Lease object in the API Server:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  holderIdentity: controller-manager-node-1 # Identity of the instance holding Leader status
  leaseDurationSeconds: 15                  # Lock lease duration before it expires
  acquireTime: "2026-06-16T22:00:00.000000Z" # When the lease was first claimed
  renewTime: "2026-06-16T22:27:00.000000Z"   # When the Leader last renewed the lease (heartbeat)
  leaseTransitions: 3                        # How many times leadership changed hands

Failover Scenario Analysis and Its Impact #

The leadership management lifecycle in the cluster follows these rules:

1. Initial Acquisition: When the cluster first starts, all Controller Manager instances try to write their identity names to the Lease/kube-controller-manager object. The first instance that successfully writes to the API Server becomes the Leader.
2. Renewal Heartbeat: The Leader must update the renewTime property in the Lease object regularly (e.g. every 2 seconds) to confirm to the cluster that it’s still healthy and working.
3. Failure Detection & Failover: Other (standby) instances periodically read the Lease object. If the Leader fails (e.g. the OS hangs or the binary crashes) and doesn’t renew the lease within leaseDurationSeconds (default 15 seconds), the standby instances detect the lease has expired. They then compete to write their identities to the Lease object. The instance that succeeds in updating first takes over leadership and immediately activates all its internal control loops.
4. Latency Tolerance Impact: During the leadership transition (about 15-30 seconds), no controller is actively running in the cluster. That means if a Pod dies in this window, a new Pod won’t be created immediately. However, this transition doesn’t disturb applications already actively running on Worker Nodes. This proves the reliability of Kubernetes’ eventual consistency architecture.

The Operator Pattern and Writing Custom Controllers #

Kubernetes’ main power isn’t limited to managing built-in resources like Pods and Services. Kubernetes is designed to be extended without limits by users through the Operator Pattern.

An Operator is a combination of two main elements:

1. Custom Resource Definition (CRD): Adds new data type schemas to the Kubernetes API Server database (e.g. a PostgresCluster or RedisSentinel data type).
2. Custom Controller: A standalone application binary we write to run a dedicated reconciliation loop managing the lifecycle of that custom resource.

Custom Controller Internal Architecture #

Writing an efficient custom controller requires understanding how it interacts with the API Server without burdening the cluster. Here’s the industry-standard architecture used by frameworks like controller-runtime (Kubebuilder and Operator SDK):

flowchart LR
    subgraph APIServerGroup["Kubernetes Control Plane"]
        APIServer["API Server"] <--> etcd[("etcd DB")]
    end

    subgraph CustomControllerProcess["Custom Controller Process (Operator)"]
        direction TB
        Reflector["Reflector"] --> FIFO["DeltaFIFO Queue"]
        FIFO --> Informer["Informer (Indexer)"]
        Informer --> Cache[("Local Cache (Lister)")]
        Informer -->|Send Event| EventHandlers["Event Handlers (Add/Update/Delete)"]
        EventHandlers -->|Enqueue Key| WorkQueue["WorkQueue"]
        
        WorkQueue -->|Take Work| Workers["Worker Threads"]
        Workers -->|1. Read Spec| Cache
        Workers -->|2. Reconcile & Act| APIServer
    end

    APIServer -.->|Watch Stream| Reflector

Operator Architecture Components: #

• Reflector: Opens a persistent Watch connection to the API Server to listen for specific changes to custom resources. Every time something changes, the Reflector pushes that data into the internal DeltaFIFO queue.
• Informer & Indexer: Reads data from DeltaFIFO, decodes it, then stores it in the Local Cache (so the controller doesn’t need to keep calling the API Server to read data — it reads from local memory via the Lister). The Informer then triggers Event Handler functions (like OnAdd, OnUpdate, OnDelete).
• WorkQueue: Event handlers don’t execute reconciliation directly. They only push the resource’s unique identifier key (e.g. "namespace/resource-name" format) into the WorkQueue. This queue has smart features like rate-limiting (limiting retry frequency during consecutive errors) and deduplication (merging multiple rapid change events of the same object into a single reconciliation task).
• Worker Threads: Parallel worker threads. They take keys from the WorkQueue, read the latest object spec from the Local Cache, compare it with the real-world condition (e.g. checking whether the physical cloud database has been created), then make external API calls (to a cloud provider API or the API Server) to reconcile the state.

Anti-Patterns in Controller Management #

When designing production systems or writing Kubernetes automation, several fatal mistakes are often made due to a lack of understanding of how asynchronous controller systems work.

Anti-Pattern 1: Bypassing Controllers with Manual or Imperative Modifications #

Trying to force a distributed system to do something instantly by ignoring the built-in controllers.

ANTI-PATTERN: Force-Deleting Pods Without Regard for the Parent ReplicaSet
// WHAT WE DO:
- Run an application under Deployment/ReplicaSet control.
- When the application misbehaves, we immediately delete the Pod manually with the force flag:
  kubectl delete pod my-app-7f4d9b-abcde --force --grace-period=0
- Or manually modify the iptables IPs directly on the Worker Node host OS to change traffic routes.

// THE CONSEQUENCES IN PRODUCTION:
- Resource Leaks: Force deletion cuts off the Garbage Collector's clean teardown cycle.
  External objects like dynamic persistent volumes or cloud load balancers can be left behind as orphans, causing cost bloat.
- Network Inconsistency: Manual iptables modifications get overwritten and removed within seconds
  by kube-proxy because the EndpointSlice Controller detects the real state doesn't match the etcd database.

✓ THE RIGHT SOLUTION:
- Always change cluster state through declarative manifest modifications (changing the Deployment spec, Service, etc.).
- Let controllers perform resource deletion and creation asynchronously and in an orderly fashion.
- To restart an application safely, use the official command that triggers a controlled rolling update cycle:
  kubectl rollout restart deployment my-app

Anti-Pattern 2: Writing Custom Controllers Without Optimistic Concurrency Control (OCC) #

A fatal mistake when writing custom Operators that causes conflicting data writes or race conditions.

// ANTI-PATTERN: Updating Objects Without Checking the ResourceVersion
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var dbCluster myv1.DatabaseCluster
    r.Get(ctx, req.NamespacedName, &dbCluster) // Read data from the local cache

    // ✗ We modify the status directly without considering changes in etcd
    dbCluster.Status.Phase = "Running"
    
    // ✗ This update fails if another component changed this object in etcd a few milliseconds ago
    r.Status().Update(ctx, &dbCluster) 
    return ctrl.Result{}, nil
}

// ✓ THE RIGHT SOLUTION: Applying a Retry Loop on OCC Conflicts
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Use Kubernetes' built-in retry library to handle resourceVersion conflicts gracefully
    err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
        var dbCluster myv1.DatabaseCluster
        if err := r.Get(ctx, req.NamespacedName, &dbCluster); err != nil {
            return err
        }
        
        dbCluster.Status.Phase = "Running"
        return r.Status().Update(ctx, &dbCluster)
    })
    
    if err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

Summary #

• The Heart of the Declarative Cluster — The Controller Manager is a single process consolidating dozens of independent control loops to keep the cluster’s real state aligned with YAML manifests.
• The Reconciliation Cycle Formula — Every controller runs an endless loop: Observe ➔ Compare ➔ Act. This design guarantees idempotency and high cluster fault tolerance.
• Controller Task Division — Cluster tasks are delegated specifically: the Node Controller monitors server health, the ReplicaSet Controller maintains Pod counts, and the Deployment Controller leads downtime-free update processes.
• Scalability via EndpointSlice — The EndpointSlice Controller splits Service IP lists into small chunks to prevent network performance degradation on large-scale clusters.
• Leader Election coordination.k8s.io — Using Lease objects as distributed locks guarantees only one active Controller Manager instance in High Availability scenarios, avoiding split-brain risk.
• Efficient Operator Architecture — Writing custom controllers requires the Informer, Lister Cache, and WorkQueue pattern to minimize direct API Server request load.
• Optimistic Concurrency Control (OCC) — Always use the RetryOnConflict pattern when writing controller code to handle parallel data write conflicts in the etcd database.

← Previous: Scheduler   Next: Etcd & Cluster Consistency →