Resource Request & Limit #

In a dense Kubernetes cluster, compute capacity management is a main pillar of system stability. Without clear limits, one container with a memory leak or CPU load spike can easily take down all other containers on the same worker node. To prevent this operational anarchy, Kubernetes provides a resource declaration mechanism through two key parameters: Resource Requests and Resource Limits.

However, writing numbers in a YAML manifest is only the first step. The implications of that declaration run deep, affecting how the scheduler works on the control plane all the way down to low-level Linux kernel interactions inside the worker node. This article covers the semantic differences between requests and limits, how the Linux kernel (through cgroups) controls CPU and RAM, how to determine accurate values from empirical data, and resource governance at the namespace level using LimitRange and ResourceQuota.

Semantic Differences: The Scheduler Contract vs the Runtime Contract #

Many beginner DevOps teams equate requests and limits, but the two are completely different contracts operating at different Pod lifecycle phases.

Here’s an essential comparison table between Resource Requests and Resource Limits:

To understand how both parameters work end-to-end, let’s look at this resource allocation lifecycle diagram:

flowchart TD
    PodCreate["New Pod Declared"] --> SpecRead["Read spec.resources"]
    
    subgraph SchedulerPhase["Stage 1: Scheduling"]
        SpecRead --> CheckRequest["Evaluate CPU & RAM 'requests'"]
        CheckRequest --> FilterNodes["Filter Nodes: Remaining allocatable capacity >= request"]
        FilterNodes --> MatchNode["Place the Pod on the Chosen Node"]
    end
    
    subgraph RuntimePhase["Stage 2: Runtime (Cgroups & Kernel)"]
        MatchNode --> RunContainer["Run the Container via Containerd"]
        RunContainer --> ApplyCgroups["Apply limits via Linux Cgroups"]
        
        ApplyCgroups --> CPUMonitor{"Is the container using\nCPU > limit?"}
        CPUMonitor -- "Yes" --> CPUThrottle["Throttle CPU (CFS Quota) -. App Slows Down .-> Container Stays Alive"]
        CPUMonitor -- "No" --> CPUNormal["CPU Running Normally"]
        
        ApplyCgroups --> RAMMonitor{"Is the container using\nRAM > limit?"}
        RAMMonitor -- "Yes" --> RAMKill["Kernel OOM-Killer Activates -. Exit Code 137 .-> Container Dies (OOMKilled)"]
        RAMMonitor -- "No" --> RAMNormal["RAM Running Normally"]
    end

The CPU Mechanism: A Compressible Resource #

CPU in Kubernetes is categorized as a compressible resource. That means if our container tries to use CPU beyond its allowed capacity, the Linux kernel doesn’t kill the container process. Instead, the kernel limits or “presses” the CPU time allotment for that container, slowing the application down.

1. CPU Requests and Linux cpu.shares #

Under the hood, when we set a CPU request value (e.g. cpu: "250m" or 0.25 cores), Kubernetes translates this into a cpu.shares weight unit on the Linux cgroups system.

• By standard, 1 CPU Core (1000m) equals 1024 shares.
• So, cpu: "250m" translates to 256 shares.

cpu.shares is a proportional allotment that’s only enforced when all physical CPU on the worker node is being contested at 100% load.

• If the cluster is quiet, a container requesting only 250m is allowed to use up to 100% of the node’s CPU if needed.
• However, if all Pods on the node suddenly get busy, the Linux kernel divides CPU cycles fairly based on each one’s shares proportion. A Pod with a 1000m CPU request is guaranteed 4 times more CPU than a Pod requesting 250m.

2. CPU Limits and CFS Quota #

Unlike requests, CPU limits are translated using the CFS (Completely Fair Scheduler) Bandwidth Control mechanism in the Linux kernel through the cpu.cfs_quota_us and cpu.cfs_period_us parameters.

• Period (cfs_period_us): The evaluation time window duration (default: 100,000 microseconds or 100 ms).
• Quota (cfs_quota_us): The active CPU time allotment the container is allowed to use within that period window.

If we set limits.cpu: "0.5" (half a core), cgroups gets configured with: $$\text{period} = 100\text{ ms}$$ $$\text{quota} = 50\text{ ms}$$

That means within every 100 ms, our container may only run for a total of 50 ms. If our container uses multi-threading and burns through its 50 ms allotment in the first 10 ms (e.g. by using 5 threads simultaneously), then for the remaining 90 ms the container process gets throttled (temporarily paused) by the kernel.

The Danger of CPU Throttling for Latency-Critical Applications #

CPU Throttling is a hidden killer of application performance. Its symptoms include:

• Sudden API latency spikes.
• Failing HTTP Readiness Probes because the container responds too slowly to Kubelet pings.
• TCP connection queue buildup.

We can monitor this throttling level from the Prometheus dashboard with the metric:

sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (container, pod)

The Memory Mechanism: An Incompressible Resource #

Physical memory (RAM) is categorized as an incompressible resource. If a process needs 1 more byte of memory to allocate a new variable, we can’t squeeze that memory; it must physically exist. If physical memory runs out, the Linux kernel is forced to take extreme action.

1. Memory Requests #

Memory requests are used by the kube-scheduler to calculate remaining worker node capacity during the scheduling phase.

• At the node kernel level, memory requests don’t set any hard runtime restriction on cgroups (there’s no direct cpu.shares equivalent for memory in cgroups v1).
• However, on modern Kubernetes using cgroups v2, memory requests translate to memory.low or memory.min values, providing page-reclaim protection so important Pod memory isn’t easily reclaimed by the kernel during memory pressure.

2. Memory Limits and the OOMKilled Disaster #

When we set limits.memory: "512Mi", cgroups writes this hard limit value to the memory.limit_in_bytes parameter file.

As soon as the container tries to allocate RAM beyond the 512MiB limit:

1. Cgroup OOM Trigger: The cgroup memory controller subsystem detects the limit violation.
2. Kernel OOM-Killer: The Linux kernel immediately activates the Out of Memory (OOM) Killer module specific to that cgroup.
3. Process Killed: The kernel picks the main process inside the container consuming the most RAM and kills it instantly with the SIGKILL signal.
4. Exit Code 137: The container dies suddenly, and the Kubelet records its status as OOMKilled with exit code 137 (which results from the system signal code $128 + 9 \text{ (SIGKILL)} = 137$).

# Check whether any Pods were recently OOMKilled
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilled

[!CAUTION] Setting memory limits too low on JVM-based applications (like Java Spring Boot) is an instant recipe for production failure. The JVM needs extra space beyond the main memory heap (Java Heap) for Metaspace, thread stacks, garbage collection overhead, and off-heap JNI memory. If we set -Xmx512m (512MB max heap), we must set the Kubernetes memory limit to at least 768MB or 1GB so the JVM process isn’t killed by the Kernel OOM-Killer.

Resource Governance at the Namespace Level #

To prevent one team or one application from sabotaging the entire cluster capacity through careless manifest writing, we must set guardrails at the namespace level using the LimitRange and ResourceQuota objects.

1. LimitRange: Enforcing Container Defaults and Bounds #

LimitRange is a policy object that automatically injects default request/limit values into containers that don’t define them, and rejects Pod creation if the values violate the minimum/maximum thresholds we allow.

Here’s a standard LimitRange manifest for a production namespace:

apiVersion: v1
kind: LimitRange
metadata:
  name: production-limit-range
  namespace: production
spec:
  limits:
  - type: Container
    # If the developer doesn't write spec.resources, these values are injected automatically:
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"
    # Reject Pod creation if it jumps past the maximum:
    max:
      cpu: "2000m"
      memory: "4Gi"
    # Reject Pod creation if it's below the minimum:
    min:
      cpu: "100m"
      memory: "128Mi"

2. ResourceQuota: Limiting Total Namespace Accumulation #

If LimitRange governs the size of each container, ResourceQuota limits the total accumulated consumption of all objects running in that namespace.

Here’s a ResourceQuota manifest to prevent one team from monopolizing the cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-resource-quota
  namespace: production
spec:
  hard:
    # Limit the total accumulated requests from all Pods:
    requests.cpu: "20"
    requests.memory: "40Gi"
    # Limit the total accumulated limits from all Pods:
    limits.cpu: "40"
    limits.memory: "80Gi"
    # Limit the number of physical objects that can be created:
    pods: "50"
    services: "25"
    persistentvolumeclaims: "10"

If the production namespace already has 19 cores of accumulated CPU requests, and a team tries to deploy a new Pod requesting 2 CPU cores, the API Server immediately rejects the request with the error message: Forbidden: exceeded quota.

Practical Guide to Determining Requests & Limits Values #

Determining resource values must not be guesswork. We must use empirical data to strike a balance between cost efficiency and operational reliability.

Step 1: Collect Actual Usage Data #

Run our workload in a staging environment with load testing representing production peak traffic. Use basic CLI commands to see real consumption:

kubectl top pods -n production --containers

For long-term analysis, look at Prometheus graphs for memory metrics:

container_memory_working_set_bytes{container="container-name", pod="pod-name"}

[!TIP] Always use the container_memory_working_set_bytes metric to calculate container RAM needs, not container_memory_rss. The Working Set metric includes RSS memory plus active memory cache that the kernel can’t release. This is the number the OOM-killer uses as its reference to kill containers.

Step 2: Use Empirical Resource Determination Formulas #

Once we have the Average and Peak usage profiles from load testing, apply these formulas:

A. CPU Calculation #

CPU requests are set low for efficiency, but limits are set loose so the application can burst quickly during startup or request spikes: $$\text{CPU Requests} = \text{CPU Usage (Average)}$$ $$\text{CPU Limits} = \text{CPU Usage (Peak)} \times 1.5 \text{ to } 3.0$$

B. Memory Calculation #

Because memory is an incompressible resource that kills Pods when exceeded, we must be conservative: $$\text{Memory Requests} = \text{Memory Usage (Average)} \times 1.1$$ $$\text{Memory Limits} = \text{Memory Usage (Peak)} \times 1.25 \text{ to } 1.5$$

Step 3: Use the Vertical Pod Autoscaler (VPA) #

Instead of calculating manually, we can leverage the Vertical Pod Autoscaler (VPA) in Recommendation mode to analyze application behavior automatically:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa-recommender
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-app
  updatePolicy:
    updateMode: "Off" # Only gives recommendations in status, doesn't force-change Pods

After running for a few days, run kubectl describe vpa api-vpa-recommender to see the ideal resource recommendations.

Anti-Patterns in Resource Management & Their Solutions #

Here are the three resource configuration mistakes that most often damage production cluster stability:

Anti-Pattern 1: Not Setting Resource Requests & Limits at All #

Deploying Pod manifests without a resources block because you assume Kubernetes magically handles everything.

ANTI-PATTERN: Deploying an Application Pod Without a resources Block
// WHAT WE DO:
- Write a web API Deployment manifest without defining requests or limits.

// THE CONSEQUENCES IN PRODUCTION:
- Lowest QoS Class: The Pod is automatically categorized as the **BestEffort** QoS class.
- Eviction Vulnerability: When the worker node faces even slight resource pressure,
  BestEffort Pods are the first victims force-evicted by the Kubelet.
- Resource Monopoly: If our container has a memory leak, it keeps eating RAM
  up to 100% of the physical worker node capacity, killing all other healthy Pods on that node.

✓ THE RIGHT SOLUTION:
- Always enforce the rule that no container launches without a resources declaration.
- Install a LimitRange in every production namespace to inject default values
  if developers forget to write the resource spec.

Anti-Pattern 2: Blindly Equalizing CPU Requests and CPU Limits (Guaranteed QoS) #

Equalizing CPU request and limit values across all microservices for YAML convenience.

ANTI-PATTERN: Setting requests.cpu: "1" and limits.cpu: "1" for All Microservices
// WHAT WE DO:
- Equalize CPU request and limit at 1 Core on all stateless application Deployments.

// THE CONSEQUENCES IN PRODUCTION:
- Very Low Cluster Utilization: The Scheduler permanently locks 1 physical CPU Core for
  each Pod. If we have 50 Pods, we need 50 physical cores in the cluster. Yet the real average
  usage of all those Pods might only be 0.05 cores. We waste up to 90% of cloud costs needlessly.
- Startup Obstacles: Web applications (especially Node.js and Java) need high CPU bursts
  during initial bootstrap. Without limit headroom above the request, the startup process
  gets badly throttled, making Pods slow to start or hang during startup.

✓ THE RIGHT SOLUTION:
- Give loose CPU burst headroom. Set CPU requests per the average runtime needs
  (e.g. `requests.cpu: "100m"`), and give higher limits (e.g. `limits.cpu: "500m"` to `"1000m"`).
- Reserve the Guaranteed QoS pattern only for stateful database applications or
  microsecond-latency apps that can't tolerate any context switching.

Anti-Pattern 3: Calculating Memory Limits Too Tight Without Runtime Buffer #

Setting the container memory limit exactly equal to the programming language’s internal heap bound.

ANTI-PATTERN: Setting Java Runtime -Xmx512m and limits.memory: "512Mi"
// WHAT WE DO:
- Cap the Java (JVM) maximum heap at 512MB in the environment variables.
- Set the Kubernetes Pod memory limit to 512MiB.

// THE CONSEQUENCES IN PRODUCTION:
- Repeated OOMKilled: The JVM needs extra memory beyond the heap to run its internal engine
  (Class metadata, thread stacks, garbage collection memory, compilation cache). The real total memory
  consumption of that Java process in the OS reaches ~650MB.
- Because our limit is locked at 512MB, the Linux kernel kills our Spring Boot container
  instantly (OOMKilled) before the application manages to serve its first HTTP request.

✓ THE RIGHT SOLUTION:
- Always provide at least 25% to 40% memory buffer beyond the programming language's heap memory limit.
- If the JVM heap is set to 512MB, set the Kubernetes memory limit to at least 768MiB or 1GiB.
- Monitor memory metrics periodically and adjust this buffer based on real working set bytes data.

Summary #

• Requests for Scheduling — requests values are used exclusively by kube-scheduler to filter and place Pods on worker nodes with sufficient remaining capacity.
• Limits for Runtime Restriction — limits values are forcibly enforced at the worker node level by Linux kernel cgroups while the container runs.
• Throttling vs OOMKilled — Exceeding CPU limits results in processing speed restriction (throttling), while exceeding memory limits results in instant killing (OOMKilled).
• Use Working Set Bytes — When analyzing application RAM needs via Prometheus, always make container_memory_working_set_bytes the primary reference metric.
• Enforce LimitRange — Use LimitRange to guarantee all cluster containers automatically get default resource values and prevent BestEffort failures in production.
• ResourceQuota for Cluster Protection — Apply ResourceQuota at the namespace level to limit teams’ total consumption and prevent cluster compute resource monopolies.
• Give a Runtime Memory Buffer — Always provide enough memory buffer beyond the runtime heap capacity (like JVM heap or Node.js memory limits) to avoid OOMKilled disasters.

← Previous: Scheduler Workflow   Next: QoS Class →