Scheduler #

In a Kubernetes cluster, every time a new Pod is declared, it isn’t immediately tied to any server machine (node). The Pod sits in an unscheduled state, queuing for a suitable home. The placement-broker component responsible for analyzing, filtering, scoring, and finally assigning the Pod to the best Worker Node in the cluster is kube-scheduler (the Scheduler).

The Scheduler acts as the cluster’s spatial architect. Without a Scheduler, all our application containers would stay stuck in Pending status forever. For developer and DevOps teams, understanding the Scheduler’s internal algorithm is crucial for designing cross-data-center high availability, efficiently allocating dedicated GPU servers, and diagnosing why a Pod fails to be placed and gets stuck in Pending status.

The Two Main Scheduling Phases: Filtering and Scoring #

When the Scheduler detects a new Pod without a placement assignment (spec.nodeName is empty), it takes that Pod from the scheduling queue. The Scheduler then runs a very strict evaluation algorithm through two main phases: Filtering and Scoring.

Here’s a visualization of the scheduling pipeline from the moment a Pod is detected until the decision is permanently written:

flowchart TD
    PodQueue["New Pod in the Scheduling Queue"] --> Fetch["Scheduler Takes the Pod"]
    Fetch --> FilterPhase["1. Filtering Phase (Predicates)\nEliminate unqualified nodes"]
    
    FilterPhase --> CheckQualified{Are there any\nqualified nodes?}
    
    CheckQualified -- "No" --> PendingState["Pod Stuck in Pending Status\n(Event: FailedScheduling)"]
    CheckQualified -- "Yes" --> ScorePhase["2. Scoring Phase (Priorities)\nScore the remaining qualified nodes (0-100)"]
    
    ScorePhase --> SelectBest["Pick the Node with the Highest Score"]
    
    SelectBest --> Binding["3. Binding Phase\nWrite the placement decision to the API Server"]
    Binding --> Success["The Kubelet on the Chosen Node Runs the Pod"]
    
    PendingState -->|Retry Loop| Fetch

1. Filtering Phase (Predicates / Eligibility Criteria) #

In the filtering phase, the Scheduler screens all Worker Nodes in the cluster to find which ones have the capacity and physical capability to run the Pod. Each filtering criterion is called a Predicate. If a node fails even one Predicate rule, it’s immediately crossed off the eligibility list.

Some of the most important built-in Predicate filters include:

  • NodeResourcesFit: Checks whether the node’s available CPU, RAM, and ephemeral-storage capacity can still accommodate the Pod’s declared resource requests.
  • NodeName: Checks whether the developer explicitly requested a specific node name in the manifest using the spec.nodeName field.
  • NodePorts: Checks whether the host network port (hostPort) requested by the Pod’s containers is already used by another container on that node. If there’s a port conflict, the node is eliminated.
  • PodTopologySpread: Checks whether placing the Pod on that node violates the cluster topology spread rules.
  • NodeAffinity: Checks whether the physical labels attached to the node match the node selection rules (Node Selector/Affinity) declared in the Pod manifest.

If the filtering phase produces zero qualified nodes, the Pod gets stranded in Pending status and the Scheduler writes a FailedScheduling event log to the API Server. The Pod stays there until new node resources become available (for example through cluster autoscaling).

2. Scoring Phase (Priorities / Evaluation Criteria) #

After getting the list of nodes that passed the filtering phase, the Scheduler scores each one to find the best-quality node. This phase is called scoring, where each qualified node receives a score from 0 to 100 based on a set of priority rules (Priorities).

Some of the priority scoring parameters include:

  • ImageLocalityPriority: Nodes that have already downloaded (cached) the container image the Pod needs get a much higher score. This greatly speeds up application boot time because the node doesn’t waste time pulling a giant image from the internet.
  • NodeAffinityPriority: Gives a higher score to nodes satisfying non-mandatory label selection preferences (preferredDuringScheduling).
  • ResourceBalancedAllocation: Evaluates the CPU/RAM usage balance ratio on each node. The Scheduler prefers nodes with balanced resource utilization (e.g. 50% CPU and 50% RAM used) over nodes with 90% CPU load but completely idle RAM.
  • SelectorSpreadPriority: Tries to spread Pods under the same ReplicaSet or Service across different nodes so they don’t cluster on one node, minimizing failure impact if that node dies.

The score from each priority rule is multiplied by its weight, then accumulated. The node with the highest total score wins. If several nodes tie exactly, the Scheduler picks one randomly. Once chosen, the Scheduler runs the Binding phase by sending a binding object to the API Server to write the chosen node name into the Pod’s spec.nodeName field.

Advanced Scheduling Patterns #

Kubernetes gives us limitless flexibility to control Pod placement precisely in production through the following features:

1. Node Affinity & Anti-Affinity #

Node Affinity lets us bind Pods to specific nodes based on labels attached to the nodes. There are two types of rules:

  • Hard Affinity (requiredDuringSchedulingIgnoredDuringExecution): A mandatory rule that must be satisfied in the filtering phase. If not met, the Pod fails to schedule.
  • Soft Affinity (preferredDuringSchedulingIgnoredDuringExecution): A priority preference in the scoring phase. If satisfied, the score increases; if not, the Pod can still run on another node.

Example of a mandatory Node Affinity declaration:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - ap-southeast-1a # The Pod must be placed in Availability Zone 1a

2. Pod Affinity & Anti-Affinity #

Unlike Node Affinity, which selects on node labels, Pod Affinity and Anti-Affinity match labels on other Pods already running in the cluster.

  • Pod Affinity: Places our Pod close to other Pods (e.g. putting a frontend Pod on the same node or zone as a Redis Cache Pod to reduce network latency).
  • Pod Anti-Affinity: Keeps our Pod away from similar Pods (e.g. forbidding two payment application Pod replicas from running on the same node or Availability Zone). This is mandatory in production to guarantee physical system redundancy.
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - payment-api
        topologyKey: topology.kubernetes.io/zone # Guarantees payment-api Pods spread across different zones

3. Taints & Tolerations #

If affinity attracts Pods to nodes, Taints and Tolerations work the opposite way — they reject Pods from arbitrarily occupying certain nodes.

  • Taint: Applied to Node objects (e.g. a GPU server gets the taint sku=gpu:NoSchedule). A tainted node rejects all Pods unless a Pod carries a matching toleration.
  • Toleration: Declared in the Pod manifest to state permission to pass that node’s taint.

There are three Taint effect types:

  1. NoSchedule: Pods without a matching toleration will never be scheduled on this node.
  2. PreferNoSchedule: The Scheduler tries to avoid scheduling on this node, but if all other nodes are full, Pods may still be placed here.
  3. NoExecute: If a node gets this taint dynamically, all Pods without a toleration currently running on it are immediately evicted.

4. Topology Spread Constraints #

Used to spread Pod replicas proportionally and evenly across topology domains (like availability zones, regions, or server racks) to prevent load piling up at a single point.

spec:
  topologySpreadConstraints:
  - maxSkew: 1                                 # Maximum Pod count difference between zones is 1
    topologyKey: topology.kubernetes.io/zone   # The distribution domain is the Availability Zone
    whenUnsatisfiable: DoNotSchedule           # If it can't be evened out, don't schedule
    labelSelector:
      matchLabels:
        app: web-server

Pod Disruption Budget (PDB) in Production #

During daily operations, cluster administrators often have to perform node maintenance (e.g. upgrading the host OS kernel). This maintenance is done with kubectl drain <node>, which forcibly evicts all Pods from the node so it can be safely shut down.

To ensure these system maintenance actions don’t damage our service availability to users (causing unintentional downtime), we must define a Pod Disruption Budget (PDB). A PDB acts as an internal service level agreement (SLA) limiting how many Pods may die simultaneously during maintenance.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
  namespace: production
spec:
  minAvailable: 2 # At least 2 healthy, active payment-api Pods must exist in the cluster at all times
  selector:
    matchLabels:
      app: payment-api

With the PDB configuration above, if an administrator runs kubectl drain, the API Server coordinates with the Scheduler to reject Pod deletion if that action would drop the active payment-api replica count below 2. The drain command stays blocked until a new replica successfully stands up on another Worker Node.

Anti-Patterns in Pod Scheduling #

Scheduling configuration mistakes that often cause serious performance issues, up to cluster death.

Anti-Pattern 1: Ignoring Resource Requests (Scheduling Starvation) #

Deploying applications to a cluster without declaring the minimum resources they need.

ANTI-PATTERN: Deploying Pods Without Filling the resources.requests Field
// WHAT WE DO:
- Create a Deployment manifest for 10 replicas of a java microservice application.
- In the container template column, we skip filling in the `resources.requests` property.

// THE CONSEQUENCES IN PRODUCTION:
- Blind Scheduling: Because requests are empty, the Scheduler assumes the Pod needs 0 CPU and 0 RAM.
- The Scheduler piles all 10 Pods onto the same Worker Node because it assumes the node is still empty.
- Once the containers actually boot and consume physical memory, the host server's RAM runs out instantly.
- The Linux kernel OOM (Out Of Memory) Killer activates and starts killing Java processes randomly.
- The node crashes completely, triggering mass panic in the cluster because all applications on it die at once.

✓ THE RIGHT SOLUTION:
- Always do load testing to measure your application's real compute needs.
- Write those minimum requirements accurately in the `resources.requests` field:
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "500m"
- This guides the Scheduler to distribute workloads evenly across all available nodes.

Anti-Pattern 2: Overusing Hard (Required) Pod Anti-Affinity #

Applying strict isolation rules that extremely limit cluster flexibility.

ANTI-PATTERN: Using requiredDuringScheduling for All Anti-Affinity Rules
// WHAT WE DO:
- Configure 5 API Pod replicas with `requiredDuringSchedulingIgnoredDuringExecution` Pod Anti-Affinity
  based on `topologyKey: kubernetes.io/hostname`.
- Our cluster currently only has 3 physical Worker Nodes.

// THE CONSEQUENCES IN PRODUCTION:
- Scheduling Deadlock: The Scheduler only manages to place 3 Pods (1 Pod per node).
  2 remaining Pods stay stuck in `Pending` status forever because no 4th or 5th physical node exists.
- Recovery Failure: If one node crashes, the dead Pod can never be rescheduled
  to the remaining nodes because it violates the mandatory anti-affinity rule. Our application redundancy actually drops drastically.

✓ THE RIGHT SOLUTION:
- Use Soft rules (`preferredDuringSchedulingIgnoredDuringExecution`) for hostname anti-affinity.
- This tells the Scheduler to try keeping Pods apart, but if forced
  (e.g. another server dies or the cluster is full), it's still allowed to place several Pods on the same node.

Summary #

  • The Cluster’s Placement Broker — The Scheduler analyzes new Pods without node assignments and determines the best Worker Node based on resources and policy constraints.
  • Filtering & Scoring Pipeline — The scheduling process is split in two: Filtering (removing unqualified nodes via Predicates) and Scoring (giving 0-100 scores to pick the best node via Priorities).
  • Redundancy via Anti-Affinity — Use Pod Anti-Affinity with zone availability topology to guarantee application replicas spread across data centers for high availability.
  • Node Protection with Taints — Use Taints on Nodes combined with Tolerations on Pods to secure special nodes (like GPU) from being entered by regular containers.
  • Maintenance Tolerance Limits — Define a Pod Disruption Budget (PDB) with minAvailable or maxUnavailable values to protect applications from downtime during host server maintenance.
  • Avoid Starvation — Always declare resources.requests values in every application manifest to guide the Scheduler in distributing load proportionally.
  • Use Soft Affinity — Avoid overusing Hard Anti-Affinity to prevent Pods getting stuck in Pending status when the cluster lacks enough physical nodes.

← Previous: API Server   Next: Controller Manager →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact