Scheduler Workflow #

In a Kubernetes cluster, placing workloads on the most appropriate hardware is a key determinant of our infrastructure’s stability, performance, and cost efficiency. The component carrying this huge responsibility is kube-scheduler. It acts as the decision-making brain that constantly watches for new Pods without nodes, evaluates the cluster’s entire physical capacity, and designates the best worker node to run those workloads.

But how is this scheduling decision actually made behind the scenes? How does kube-scheduler sort through hundreds of nodes in mere milliseconds? And most importantly, what should we do when our Pod gets stuck in Pending status in production? This article fully dissects the kube-scheduler workflow end-to-end, its internal queue architecture, the details of the filtering and scoring phases, the preemption mechanism, and tactical guides for diagnosing scheduling problems.

End-to-End Workflow: From YAML to Active Container #

Scheduling a Pod isn’t a single instant operation — it’s a series of distributed interactions involving various cluster components. We need to understand this lifecycle to identify bottlenecks precisely.

Here’s the process sequence from submitting a YAML manifest until the container runs on a worker node:

flowchart TD
    PodArrival["New Pod Detected (status: Pending, nodeName: '')"] --> Queue["Scheduling Queue (activeQ)"]
    Queue --> Dequeue["Take the Highest Priority Pod"]
    
    subgraph FilteringPhase["Phase 1: Filtering (Predicates)"]
        Dequeue --> FilterStart["Evaluate All Cluster Nodes in Parallel"]
        FilterStart --> Filter1["NodeUnschedulable: Check the node cordon status"]
        FilterStart --> Filter2["PodFitsResources: Check the CPU & RAM requests"]
        FilterStart --> Filter3["NodePorts: Check container port conflicts"]
        FilterStart --> Filter4["NodeAffinity & Taints: Match labels & tolerations"]
        
        Filter1 --> FilterMerge["Merge Filter Results"]
        Filter2 --> FilterMerge
        Filter3 --> FilterMerge
        Filter4 --> FilterMerge
    end
    
    FilterMerge --> CheckCandidates{"Are there any\nqualified nodes?"}
    CheckCandidates -- "No" --> Unscheduled["Put into unschedulableQ (Failed Scheduling)"]
    CheckCandidates -- "Yes" --> ScoringPhase
    
    subgraph ScoringPhase["Phase 2: Scoring (Priorities)"]
        ScoreStart["Score 0-100 on All Qualified Nodes"]
        ScoreStart --> Score1["LeastAllocated: Prioritize quiet nodes"]
        ScoreStart --> Score2["ImageLocality: Prioritize nodes with the image"]
        ScoreStart --> Score3["NodeAffinityPriority: Calculate optional affinity weight"]
        
        Score1 --> ScoreMerge["Calculate the Weighted Sum (Final Score)"]
        Score2 --> ScoreMerge
        Score3 --> ScoreMerge
    end
    
    CheckCandidates -- "Yes" --> ScoreStart
    ScoreMerge --> SelectWinner["Pick the Node with the Highest Score"]
    
    subgraph BindingPhase["Phase 3: Binding"]
        SelectWinner --> BindCreate["Create a Binding Object in the API Server"]
        BindCreate --> BindWrite["Write nodeName to the Pod Object in Etcd"]
        BindWrite --> KubeletWatch["The Chosen Node's Kubelet Detects the Watch Event"]
        KubeletWatch --> ContainerStart["Start Running the Application Container"]
    end

Explanation of the 7 Main Processing Stages: #

  1. Manifest Submission and Validation: We run kubectl apply -f deployment.yaml. The API Server receives the request, authenticates and authorizes it, and passes it through the Admission Controller. If approved, the Deployment object is stored in the etcd database.
  2. Pod Object Creation: The Deployment Controller detects the new object and creates a ReplicaSet object. The ReplicaSet Controller then creates Pod replicas per the desired count (e.g. 3). These new Pods are stored in etcd with initial Pending status and without a spec.nodeName property (an empty string "").
  3. Watch API Detection: The kube-scheduler has a persistent connection to the API Server using the Watch API mechanism. As soon as it detects a new Pod without a nodeName, it immediately puts that Pod into its own scheduling queue.
  4. Filtering Phase (Predicates): The kube-scheduler scans all nodes in the cluster and eliminates nodes that don’t meet the minimum requirements (e.g. insufficient CPU/RAM, having an untolerated taint, or violating affinity rules).
  5. Scoring Phase (Priorities): Nodes passing the filtering phase are scored using a set of scoring functions. Each criterion (like memory usage efficiency, local container image availability, etc.) gets a score from 0 to 100, multiplied by each plugin’s weight to produce a final global score per node.
  6. Optimistic Binding: The kube-scheduler picks the highest-scoring node. Before writing data to etcd, it performs optimistic binding locally in the scheduler’s memory cache to avoid collisions if other Pods are being scheduled in parallel. Once safe, it sends a Binding object to the API Server to write the Pod’s spec.nodeName in etcd.
  7. Kubelet Execution: The kubelet running on the chosen worker node periodically watches the API Server for Pods assigned to its name. Once it detects a new Pod with its nodeName, the Kubelet instructs the container runtime (like containerd) to pull the image, set up networking, mount storage volumes, and start running the container.

Internal Queue Architecture (Scheduling Queue) #

The kube-scheduler doesn’t process Pods randomly. It manages a very efficient internal queue to ensure Pods ready for scheduling are processed first, while Pods that failed scheduling don’t waste cluster CPU cycles.

This queue is divided into three main data structures:

[Kube-Scheduler Queue]
  │
  ├──► activeQ (Active Queue)
  │      Pods ready to be scheduled now. Sorted by priority.
  │
  ├──► backoffQ (Temporary Backoff Queue)
  │      Pods that just failed scheduling due to resource limits.
  │      Waiting for a backoff duration before being retried.
  │
  └──► unschedulableQ (Failed / Ineligible Queue)
         Pods that were tried but proven to have no matching node.
         Waiting for a cluster state change to retry.

1. activeQ (Active Queue) #

This is the main queue where the kube-scheduler takes the next Pod to schedule. It’s implemented as a priority heap. Pods with the highest priority value (based on the PriorityClass we assign) automatically jump to the front of the queue for processing first.

2. backoffQ (Backoff Queue) #

When a Pod fails to schedule (e.g. no node has enough memory at that moment), the Pod must not immediately return to the active queue. If it returned immediately, it would trigger an endless busy loop burdening the CPU.

Instead, the Pod moves to backoffQ. Here, the Pod must wait a certain duration (backoff period) that increases exponentially after each consecutive failure. Once the wait ends, the Pod moves back to activeQ.

3. unschedulableQ (Unschedulable Queue) #

If after several attempts the Pod still can’t find a matching node, it moves to unschedulableQ. Here, the Pod sits passively without consuming scheduler CPU cycles.

Pods in unschedulableQ are only moved back to activeQ when a cluster change event occurs that could potentially resolve their problems. These triggering events include:

  • A new worker node joining the cluster.
  • Updates or deletions of other Pods freeing up CPU/RAM resources.
  • Label changes on worker nodes.
  • Taint removal from nodes.

The Filtering Pipeline (Predicates) in Depth #

The filtering phase aims to eliminate all worker nodes that physically or logically can’t run our Pod. The kube-scheduler runs these filters in parallel to speed up the process.

Here are the core filters we must understand, because they’re often the reason Pods get stuck in Pending status:

1. PodFitsResources #

This filter checks whether the node’s remaining allocatable capacity is sufficient to meet the resource requests values (CPU, RAM, and Ephemeral Storage) declared by our Pod’s containers.

[!IMPORTANT] This filter only counts the request value, not limit. If a node has 2GiB of allocatable memory left, and our Pod requests requests: 4GiB but has limits: 8GiB, the node is immediately eliminated because the remaining physical memory can’t hold the minimum request.

2. NodeUnschedulable #

This filter detects whether the worker node is in cordon condition (spec.unschedulable: true). If we’re doing node maintenance and mark it with kubectl cordon node-1, this filter automatically eliminates node-1 from new Pod scheduling.

3. NodePorts #

If our container explicitly defines a host port using hostPort (e.g. mapping container port 80 directly to port 80 on the physical node), this filter screens worker nodes. If port 80 on a node is already used by another Pod, that node is eliminated to avoid network port conflicts.

4. NodeAffinity & PodSelector #

This filter matches the node affinity rule expressions (nodeSelector or nodeAffinity) we write in the Pod spec against the labels attached to physical worker nodes. Nodes without matching labels are immediately removed.

5. TaintToleration #

This filter evaluates whether the worker node has a taint (e.g. key=value:NoSchedule). A tainted node only accepts Pods with a matching toleration declaration. If our Pod doesn’t have one, this filter eliminates the node.

The Scoring Pipeline (Priorities) in Depth #

After the filtering phase finishes, the kube-scheduler gets a list of eligible candidate nodes. To pick the best node among the eligible ones, the scheduler scores each node in the scoring phase.

Each scoring function (Scoring Plugin) produces a value from 0 to 100. The total score is calculated using a weighted sum formula:

$$\text{Node Final Score} = \sum_{i=1}^{n} (\text{Plugin Score}_i \times \text{Plugin Weight}_i)$$

Here are some default scoring functions that play an important role in determining where our Pod is placed:

1. LeastAllocated (Default Weight: 1) #

This plugin prioritizes nodes with the most CPU/RAM remaining after subtracting the total requests of all Pods running on them. Its internal formula is:

$$\text{Score} = \frac{(\text{Capacity} - \text{Total Requests}) \times 100}{\text{Capacity}}$$

The main goal of LeastAllocated is to spread workloads evenly across all available worker nodes (resource spreading), preventing one node from getting very hot while others sit idle.

2. ImageLocality (Default Weight: 1) #

This plugin gives higher scores to worker nodes that already downloaded (cached) the container image our Pod needs locally.

  • If a 2GiB image already exists on node-1 but not on node-2, then node-1 gets a much higher ImageLocality score.
  • This is designed to speed up Pod startup by avoiding the image pull wait time from an external container registry.

3. BalancedResourceAllocation (Default Weight: 1) #

This plugin evaluates how balanced CPU and RAM usage is inside the node. It looks for nodes where the CPU and memory usage ratios stay equivalent after the Pod is placed.

  • If we place a memory-hungry Pod on a node that’s already short on memory but has abundant CPU, the ratio becomes unbalanced.
  • This plugin tries to avoid resource fragmentation, where a worker node runs completely out of RAM while leaving 90% of CPU unusable by anyone.

4. NodeAffinityPriority (Default Weight: 2) #

This plugin evaluates preferred affinity rules (soft affinity or preferredDuringSchedulingIgnoredDuringExecution). If a node meets the preferred label criteria we want, the plugin gives a high bonus score. Because this is a developer instruction, its weight is set higher than the default balancing functions.

Pod Priority and Preemption (Workload Eviction) #

In high-density production environments, we often face situations where the cluster runs completely out of physical resources. When that happens and we need to deploy a very important Pod (e.g. the main API gateway), how can that Pod still run?

Kubernetes provides the Pod Priority and Preemption feature, which lets high-priority Pods evict low-priority Pods to get resources.

1. Defining the PriorityClass Object #

First, we must create a PriorityClass object that sets the administrative priority weight value:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-api-priority
value: 1000000                     # Integer value (max 1,000,000,000)
globalDefault: false               # If true, all Pods without a priority get this value
description: "High priority for the main production API Gateway."

2. Applying the PriorityClass to the Pod Spec #

We connect this priority to our Pod spec manifest using the priorityClassName property:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: production
spec:
  replicas: 2
  template:
    spec:
      priorityClassName: critical-api-priority # Connects to the PriorityClass
      containers:
      - name: gateway-app
        image: company/gateway:v1.1
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"

How Preemption Works: #

When the high-priority api-gateway Pod enters the activeQ, the kube-scheduler tries to find a suitable node in the filtering phase. If no node has 2 CPU cores and 4GiB RAM remaining, the preemption loop activates:

flowchart TD
    NoResource["High-Priority Pod Stuck (No Resources)"] --> PreemptActive["Activate the Preemption Mechanism"]
    PreemptActive --> FindVictims["Find a Node with Lower-Priority Pods"]
    
    FindVictims --> EvaluateNode{"Does Evicting Those Pods\nFree Up Resources?"}
    EvaluateNode -- "No" --> TryNextNode["Evaluate the Next Worker Node"]
    EvaluateNode -- "Yes" --> EvictTarget["Send a Termination Signal (Evict) to Low-Priority Pods"]
    
    EvictTarget --> WaitGrace["Wait for the Grace Period (SIGKILL)"]
    WaitGrace --> ScheduleWinner["Schedule the High-Priority Pod on That Node"]
  1. Evaluate Candidate Victims: The kube-scheduler looks for a worker node where stopping its lower-priority Pods would free up enough resources for our high-priority Pod.
  2. Pick the Smallest Victim Node: The scheduler tries to minimize eviction impact. It picks the node with the smallest total Pod priority so important applications aren’t disturbed.
  3. Terminate Victim Pods: The scheduler marks the preemptor property on the high-priority Pod and sends a termination (eviction) signal to the low-priority Pods. The low-priority Pods get the normal grace period for clean shutdown.
  4. Rescheduling: After the low-priority Pods die and free memory, the scheduler places our high-priority Pod on that node. The evicted low-priority Pods return to the scheduling queue to find another node (if one exists).

Practical Guide to Diagnosing Scheduling Problems #

As a cluster administrator, we must be able to quickly distinguish whether a Pod startup failure is caused by network issues, storage, application code bugs, or purely scheduling decisions by the kube-scheduler.

Here are systematic steps to diagnose scheduling problems:

1. Check Pod Events #

The first mandatory step is checking the event history the API Server recorded on our Pod object:

kubectl describe pod <pod-name> -n <namespace>

Scroll to the very bottom of the manifest output to see the Events section. If the problem is in the kube-scheduler, we’ll see a FailedScheduling warning:

Events:
  Type     Reason             Age   From               Message
  ----     ------             ----  ----               -------
  Warning  FailedScheduling   45s   default-scheduler  0/3 nodes are available: 1 node(s) were unschedulable, 2 Insufficient cpu.

2. Read the Worker Node Resource Allocation Status #

If we get an Insufficient cpu or Insufficient memory message, we must check the remaining capacity across all physical cluster nodes:

kubectl describe nodes | grep -A 8 "Allocated resources"

This output shows the resource allocation percentage claimed by all actively running Pods:

Allocated resources:
  Resource           Requests      Limits
  --------           --------      ------
  cpu                3650m (91%)   8000m (200%)
  memory             6920Mi (86%)  12200Mi (152%)

[!NOTE] Note that the allocation percentage is calculated from the Requests value. If the CPU request allocation already reaches 91%, the kube-scheduler won’t allow new Pods requesting large CPU into that node, even if the real CPU usage at that moment is still very low (e.g. only 10%).

3. Identify Node Taints and Labels #

If the error message says node(s) had untolerated taint, check which taints are attached to our nodes:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Make sure our Pod spec has matching tolerations declarations to pass those taint restrictions.

4. Read the Internal Kube-Scheduler Logs #

For advanced tracking cases, we can read the internal logs of the kube-scheduler process itself. If we use a self-managed cluster (where the scheduler runs as a static pod on the control plane node), we can read it with:

kubectl logs -n kube-system -l component=kube-scheduler

Anti-Patterns in Scheduling Management #

Here are two fatal mistakes that often paralyze production clusters due to scheduling misconfiguration:

Anti-Pattern 1: Manually Modifying Scheduler Config Files on Managed Clusters #

Trying to change the scoring algorithm behavior by modifying binary configs on the master node of a managed cloud service.

ANTI-PATTERN: Changing the Kube-Scheduler Policy Config Manually on a GKE/EKS Control Plane
// WHAT WE DO:
- SSH into the control plane VM (master node) on a managed cloud cluster (GKE/EKS).
- Manually modify the `kube-scheduler.yaml` config file to change scoring plugin weights.

// THE CONSEQUENCES IN PRODUCTION:
- Lost Support: The cloud provider detects unauthorized control plane modifications.
- Broken Update Cycles: When the provider auto-upgrades the cluster (e.g. from 1.28 to 1.29),
  our manual config gets overwritten or even causes the upgrade process to stall completely.
- Cluster Instability: Raw scheduler config modifications without proper replication support
  can trigger split-decision in the kube-scheduler leader election.

✓ THE RIGHT SOLUTION:
- In managed cloud environments, never touch binary config files on the control plane.
- If you need custom scheduling settings, create new Kube-Scheduler Profiles
  through a KubeSchedulerConfiguration manifest, then deploy it as a companion custom scheduler (Multiple Schedulers).
- Set `schedulerName: our-custom-scheduler` on the Pod spec that should use those rules.

Anti-Pattern 2: Setting Resource Requests Too Low to Pass the Filtering Phase #

Cheating the PodFitsResources filter by lowering requests values extremely while leaving limits very high.

ANTI-PATTERN: Writing CPU Requests: 10m (0.01 Core) but Limits: 4000m (4 Cores) for a Java Web App
// WHAT WE DO:
- Because our cluster is dense and we want our Java web Pod running fast without being blocked
  by Pending status, we manipulate the Pod spec by setting a very small CPU request (10m).
- However, we know Java needs a lot of CPU at startup, so we set limits as high as 4 Cores.

// THE CONSEQUENCES IN PRODUCTION:
- Extreme Overcommitment: The kube-scheduler trusts the 10m number and piles dozens of Java Pods
  on the same worker node because it assumes the workload is very light.
- Mass CPU Throttling: When all those Java Pods start simultaneously, they fight over
  the node's physical CPU capacity. Application latency soars, liveness probes hang, and the cluster becomes unresponsive.
- Cascading OOMKills: The same behavior on memory triggers the Kubelet on that node to kill
  our containers randomly due to physical memory exhaustion (Out of Memory).

✓ THE RIGHT SOLUTION:
- Always set `resource requests` values as close as possible to the app's real CPU/RAM usage profile at runtime.
- Use the Vertical Pod Autoscaler (VPA) in staging environments to recommend accurate request numbers.
- Use LimitRange to cap the maximum request-to-limit ratio so developers can't write extreme deviations.

Summary #

  • Two-Phase Scheduling Pipeline — The kube-scheduler determines worker nodes through two main phases: Filtering (eliminating unqualified nodes) and Scoring (rating qualified nodes to find the best one).
  • Request-Based Process — Remember that the resource filter decision (PodFitsResources) only counts requests values, not container limits.
  • Smart Queue Management — The scheduler splits the queue into activeQ (ready to process), backoffQ (temporary delay), and unschedulableQ (parked failures) to save master node CPU usage.
  • Image Locality Priority — The kube-scheduler prioritizes worker nodes that already cache local container images to save image pull time from the registry.
  • Downward API for Indices — Leverage JOB_COMPLETION_INDEX to efficiently split data range processing among Pods running in parallel.
  • Preemption for Critical Workloads — Configure PriorityClass so critical production Pods can evict non-critical Pods when the cluster faces physical resource scarcity.
  • Investigate via Events — Use kubectl describe pod to read the scheduling failure messages the scheduler records in the Events section.

← Previous: Job & CronJob   Next: Resource Request & Limit →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact