QoS Class #

In a production cluster, a worker node running out of memory (node memory pressure) is an emergency scenario that will inevitably happen sooner or later. When it does, the Kubelet acting as the node’s guardian must not let the worker node crash entirely. The Kubelet must actively choose and sacrifice (evict) certain Pods to free up memory space.

How does Kubernetes determine which Pod must be killed first and which Pod must be protected at all costs? This crucial decision is based on the Quality of Service (QoS) class attached to each Pod. Kubernetes’ QoS system operates automatically under the hood, translating the resource request/limit configuration we write into container physical resilience priorities at the Linux kernel level. This article covers the three Kubernetes QoS classes in depth, the oom_score_adj calculation mechanism, the eviction flow, the modern Memory QoS era in cgroups v2, and the configuration anti-patterns we must avoid.

The Three Kubernetes QoS Classes in Depth #

Kubernetes divides workloads into three QoS categories: Guaranteed, Burstable, and BestEffort. We can’t write these QoS classes directly in the YAML manifest (there’s no spec.qosClass field). The Kubernetes system declaratively evaluates the resources block across all containers in the Pod and assigns the class automatically.

Here’s a visualization of the QoS class determination decision flow:

flowchart TD
    PodArrival["New Pod Created"] --> ReadSpec["Evaluate the spec.resources Block of All Containers"]
    
    ReadSpec --> CheckGuaranteed{"Do ALL containers define\nRequests & Limits IDENTICALLY\nfor CPU AND Memory?"}
    
    CheckGuaranteed -- "Yes" --> ClassGuaranteed["QoS Class: Guaranteed (oom_score_adj: -997)"]
    CheckGuaranteed -- "No" --> CheckBestEffort
    
    CheckBestEffort{"Do ALL containers define\nNEITHER Requests NOR Limits at all?"}
    
    CheckBestEffort -- "Yes" --> ClassBestEffort["QoS Class: BestEffort (oom_score_adj: 1000)"]
    CheckBestEffort -- "No" --> ClassBurstable["QoS Class: Burstable (oom_score_adj: 2 to 999)"]

1. Guaranteed: The Maximum Protection Class #

A Pod gets the Guaranteed class when we provide absolute resource guarantees with zero discrepancy between requests and limits.

Determination Requirements: #

  • Every container in the Pod (including init containers and sidecars) must define requests and limits for CPU.
  • Every container in the Pod must define requests and limits for Memory.
  • The request and limit values for each resource must be exactly equal (identical) and must not be zero.

Example Guaranteed Pod Manifest: #

apiVersion: v1
kind: Pod
metadata:
  name: critical-db-app
  namespace: database
spec:
  containers:
  - name: postgres
    image: postgres:15
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "2"          # Exactly equal to the CPU request
        memory: "4Gi"      # Exactly equal to the memory request
  - name: backup-sidecar
    image: backup-agent:v1
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "100m"       # The second container must also be set identically
        memory: "128Mi"

Operational Implications: #

Guaranteed Pods are the highest caste in the Kubernetes system. The Kubelet gives top priority to protecting these Pods. They’re the very last targets touched by both the eviction process and the Kernel OOM-Killer. Kubernetes only kills a Guaranteed Pod if the worker node truly runs out of other options, or if a container inside the Guaranteed Pod itself leaks memory beyond its declared limit.

2. Burstable: The Flexible Class with Dynamic Scalability #

A Pod gets the Burstable class if we define resource requests and limits but give the container room to burst beyond its requests when needed.

Determination Requirements: #

  • The Pod doesn’t qualify for the Guaranteed class (e.g. because limit values are set higher than requests).
  • At least one container in the Pod defines a memory or CPU request.

Example Burstable Pod Manifest: #

apiVersion: v1
kind: Pod
metadata:
  name: api-web-service
  namespace: production
spec:
  containers:
  - name: web-app
    image: company/web-api:v2.1
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "1000m"      # CPU limit higher than request -> Burstable
        memory: "512Mi"   # RAM limit higher than request -> Burstable

Operational Implications: #

Burstable Pods are designed to balance cluster resource usage efficiency. When the cluster has spare capacity, Burstable Pods are allowed to use CPU and RAM up to their limit. However, if the node faces memory pressure, Burstable Pods sit second in line for eviction. The Kubelet sorts Burstable Pods by how far their real memory usage exceeds the promised request number.

3. BestEffort: The Opportunistic Class Without Guarantees #

A Pod gets the BestEffort class when we give the scheduler absolutely no information about the container’s resource needs.

Determination Requirements: #

  • No container in the Pod defines requests or limits, for either CPU or memory.

Example BestEffort Pod Manifest: #

apiVersion: v1
kind: Pod
metadata:
  name: log-scraper
  namespace: tools
spec:
  containers:
  - name: scraper
    image: company/log-scraper:latest
    # No resources block at all

Operational Implications: #

BestEffort Pods are the lowest class citizens. They get no compute capacity guarantees from the cluster. The Scheduler places these Pods on any node with an empty slot, without accounting for physical RAM availability. As soon as the worker node faces even slight resource pressure, the Kubelet immediately force-kills all BestEffort Pods without mercy to save the worker node and the other Guaranteed/Burstable Pods.

The Eviction Mechanism and Kernel Priority Score (oom_score_adj) #

To understand how the Kubelet controls container kill priority when a node runs out of memory, we must look at how Kubernetes interacts with the internal Linux Out of Memory (OOM) Killer mechanism.

On Linux OS, every process has an oom_score value (kill vulnerability score) and an oom_score_adj control value (score adjuster) ranging from -1000 (must not be killed) to 1000 (first victim). The Kubelet calculates and writes the oom_score_adj value for each container based on its QoS class:

[The Kubelet's oom_score_adj Calculation Formula]

 Guaranteed:  Permanently locked at -997.
              (Only one step above critical system processes like the Kubelet / Docker daemon).

 BestEffort:  Permanently locked at 1000.
              (The first victim the Kernel executes immediately).

 Burstable:   Calculated dynamically in reverse based on request capacity:
              oom_score_adj = 1000 - (10 * % Request Against Node Memory)

Detailed Explanation of the Burstable Formula: #

If a worker node has 32GB total RAM, and our Burstable Pod requests 3.2GB of memory (equivalent to 10% of the node’s total memory), the Kubelet calculates:

$$\text{oom_score_adj} = 1000 - (10 \times 10) = 900$$

  • If another Burstable Pod on the same node requests 16GB (50% of the node’s total memory), its adjustment value is: $$\text{oom_score_adj} = 1000 - (10 \times 50) = 500$$
  • The larger the memory request a Burstable Pod makes, the lower its oom_score_adj value, meaning that Pod is more protected from the OOM Killer than Burstable Pods requesting small resources.

The Eviction Flow When a Node Faces Resource Pressure: #

When the worker node’s physical memory shrinks past the danger threshold (eviction threshold, e.g. remaining RAM < 100Mi), the Kubelet activates a rescue loop with the following eviction order:

flowchart TD
    NodeStress["Node Under Memory Pressure"] --> EvictLoop["Kubelet Activates the Eviction Loop"]
    
    subgraph EvictHierarchy["Pod Eviction Hierarchy"]
        EvictLoop --> Tier1["1. Evaluate BestEffort Pods (oom_score_adj: 1000)"]
        Tier1 --> EvictBestEffort["Evict All BestEffort Pods Immediately"]
        
        EvictBestEffort --> CheckPressure1{"Is the memory\nsafe now?"}
        CheckPressure1 -- "Yes" --> StopEvict["Stop Eviction. Node Back to Health."]
        CheckPressure1 -- "No" --> Tier2["2. Evaluate Burstable Pods (oom_score_adj: 2 to 999)"]
        
        Tier2 --> SortBurstable["Sort Burstable Pods by:\nMemory usage furthest above requests"]
        SortBurstable --> EvictBurstable["Evict the Worst Burstable Pods One by One"]
        
        EvictBurstable --> CheckPressure2{"Is the memory\nsafe now?"}
        CheckPressure2 -- "Yes" --> StopEvict
        CheckPressure2 -- "No" --> Tier3["3. Evaluate Guaranteed Pods (oom_score_adj: -997)"]
        
        Tier3 --> EvictGuaranteed["Evict Guaranteed Pods (Only If Truly Forced)\nEspecially if the Pod itself has a memory leak"]
    end

Memory QoS in the Linux Cgroups v2 Era #

On modern Kubernetes versions (1.22+), the Memory QoS feature was introduced, leveraging the cgroups v2 control subsystem on modern Linux kernels. On the old cgroups v1 architecture, memory restrictions were binary: containers either ran freely or got instantly killed by the OOM-killer when touching the limit.

Cgroups v2 introduces a much finer (gradual control) mechanism using three parameters:

  1. memory.min (Absolute Guarantee): Locks the minimum memory amount that must never be reclaimable by the kernel’s page-reclaim system under any condition. Kubernetes maps this from requests.memory for Guaranteed Pods.
  2. memory.low (Soft Protection): Provides memory protection guarantees while the system isn’t under heavy stress. If stress occurs, the kernel may slowly reclaim this memory. Kubernetes maps this from requests.memory on Burstable Pods.
  3. memory.high (Throttling Boundary): Acts as a safety valve before the container hits the hard limit. When container memory usage touches the memory.high threshold (usually auto-set around 80-90% of limits), the Linux kernel actively slows down the container’s memory allocation process (direct page reclaim throttling) instead of killing it outright. This gives the application time to do internal memory cleanup (Garbage Collection) on its own.

QoS Class Implementation Strategies in Production #

Choosing a QoS class shouldn’t be uniform across all applications. We must match the class to how critical the workload is to business operations:

1. When to Use Guaranteed QoS? #

Use the Guaranteed class only for workloads that are highly critical, latency-sensitive, and need constant stable CPU/RAM performance guarantees.

  • Example Use Cases: Main database clusters (PostgreSQL, MariaDB, Elasticsearch), main traffic entry API Gateways, or authorization/payment microservices.
  • Trade-off: Results in more expensive cloud cluster costs because Kubernetes reserves full capacity in node memory, not allowing that memory to be shared with other applications even when our app is idle.

2. When to Use Burstable QoS? #

This is the de-facto standard class for most stateless microservice applications in production.

  • Example Use Cases: Web API applications (Node.js, Go, Python), background worker applications processing interface queues, or Redis cache servers.
  • Advantage: Enables high cost efficiency through overcommitment (we can schedule total Pod limits exceeding the node’s memory capacity, assuming not all applications will spike simultaneously).

3. When to Use BestEffort QoS? #

Use this class exclusively for non-critical workloads that are opportunistic in nature and have no business impact if they suddenly die mid-way.

  • Example Use Cases: Daily data scraper scripts, non-realtime log monitor agents, or development sandbox environments.
  • Safety: Always make sure our application has retry and auto-resume mechanisms if its process stops abruptly due to eviction.

Namespace Hardening: Preventing BestEffort Pod Formation #

In multi-tenant clusters shared by many developer teams, letting developers deploy BestEffort Pods in production namespaces is a big danger.

We can automatically block BestEffort Pod formation by installing a LimitRange policy that forces the injection of default request values:

apiVersion: v1
kind: LimitRange
metadata:
  name: prevent-best-effort-policy
  namespace: production
spec:
  limits:
  - type: Container
    # If the developer doesn't write memory/CPU requests in their YAML manifest:
    # The LimitRange automatically injects these values:
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "200m"
      memory: "256Mi"

With the LimitRange above installed:

  • Every Pod deployed in the production namespace is guaranteed to be at least Burstable because it has a default request.
  • No Pod will accidentally be BestEffort, protecting the cluster from uncontrolled eviction threats.

QoS Class Anti-Patterns and Their Solutions #

Here are two fatal mistakes that often happen in production clusters due to misunderstanding QoS Class behavior:

Anti-Pattern 1: Deploying Stateful Databases Using the Burstable or BestEffort Class #

Deploying critical production databases (like PostgreSQL Master) without equalizing memory request and limit values to save namespace quota.

ANTI-PATTERN: Setting requests.memory: "2Gi" but limits.memory: "8Gi" on a PostgreSQL Master Pod
// WHAT WE DO:
- Let the PostgreSQL database Pod run as the Burstable QoS class.
- During normal traffic, the database uses about 3GB RAM (exceeding the 2GB request).

// THE CONSEQUENCES IN PRODUCTION:
- When the worker node faces memory pressure due to another Java app leaking memory,
  the Kubelet scans all Burstable Pods.
- Because PostgreSQL uses RAM beyond its request, it becomes the primary eviction target.
- The PostgreSQL database gets killed instantly mid active write transaction,
  causing database file corruption (*data corruption*) and system downtime.

✓ THE RIGHT SOLUTION:
- Always set production stateful databases as the Guaranteed QoS class.
- Strictly ensure requests.memory == limits.memory and requests.cpu == limits.cpu.

Anti-Pattern 2: Setting Sidecar Containers Without Resource Specs in a Guaranteed Pod #

Forgetting to write the resource block on sidecar containers (like logging agents or service mesh proxies), while the main container is already Guaranteed.

ANTI-PATTERN: Writing a Full resources Spec for the App Container but Leaving the Proxy Container Empty
// WHAT WE DO:
- Main container: requests = limits (Guaranteed).
- Sidecar container: no resources spec at all (BestEffort).

// THE CONSEQUENCES IN PRODUCTION:
- Automatic Class Downgrade: Kubernetes evaluates all containers in the Pod.
  Because one sidecar container has no resource request/limit,
  the entire Pod's QoS Class drops drastically from **Guaranteed** to **Burstable**.
- Our important Pod loses its maximum protection rights and can be evicted at any time
  by the Kubelet just because of a configuration oversight on the helper container.

✓ THE RIGHT SOLUTION:
- Make sure ALL containers (including helper containers) in the Pod YAML manifest have
  identical request and limit declarations without exception to maintain Guaranteed status.

Summary #

  • QoS Is Determined Automatically — The QoS class is determined entirely by Kubernetes based on the resource request and limit configuration; there’s no special parameter to manually choose the class.
  • Guaranteed (Safest) — Requires equal requests and limits values for CPU and memory across all containers; safest from node eviction threats.
  • Burstable (Flexible) — Forms when requests and limits aren’t identical; evicted based on how far real memory usage passes the request number.
  • BestEffort (Most Vulnerable) — Defines no resource requests or limits at all; becomes the first victim instantly killed when the node faces memory pressure.
  • oom_score_adj Protection — The Kubelet writes adjustment scores to the Linux kernel: Guaranteed gets near-absolute protection (-997), while BestEffort is prepared for death (1000).
  • LimitRange Hardening — Always install a LimitRange with default requests in production namespaces to prevent BestEffort Pods from slipping through and endangering node stability.
  • Sidecars Must Be Filled — Don’t leave sidecar containers empty without resources, because that destroys the main container’s Guaranteed status.

← Previous: Resource Request & Limit   Next: Ephemeral vs Persistent Storage →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact