Resource Management #
In production-level Kubernetes clusters, allocating compute resources like CPU and memory is the most important factor determining application stability and infrastructure cost efficiency. If we let application containers run without planned resource settings, we’re placing a time bomb in the cluster. Unrestricted containers can devour the Node’s entire physical memory, causing other nearby pods to die suddenly (noisy neighbor effect), or even triggering total host operating system failures (kernel panics). Conversely, over-allocating for momentary safety wastes cloud budgets because lots of paid compute capacity never gets used. Resource Management is the discipline of balancing these needs declaratively. This article deeply discusses the Requests vs Limits concepts, CPU and memory allocation characteristics, Quality of Service (QoS) classification, default policy implementation via LimitRanges, quota restrictions via ResourceQuotas, and data-driven value determination techniques based on observability data.
The Requests vs Limits Concepts #
Kubernetes divides container resource allocation definitions into two main parameters: Requests and Limits. These two parameters have very different roles in the scheduling phase and the runtime phase.
+-------------------------------------------------------------------------+
| LIMITS |
| - The upper bound of container compute absorption in the Runtime phase|
| - CPU: Throttling (CFS quota) if exceeded |
| - Memory: Pod instantly OOMKilled if exceeded |
+-------------------------------------------------------------------------+
| REQUESTS |
| - The minimum guaranteed slot available in the Scheduling phase |
| - Used by the kube-scheduler to choose physical Nodes |
| - Calculated from the "Allocatable" capacity on Nodes |
+-------------------------------------------------------------------------+
1. Resource Requests (The Scheduling Phase) #
Requests determine the minimum CPU and memory amount guaranteed to be available for the container. When we submit a Pod manifest, the kube-scheduler scans all active Nodes and calculates the remaining allocatable compute capacity (Allocatable Capacity).
The Scheduler only places Pods on Nodes with remaining capacity greater than or equal to the Pod’s total Requests. If no Node can meet the Requests requirements, the Pod gets stuck in the Pending status with Insufficient CPU or Insufficient Memory error messages.
[!IMPORTANT] Requests aren’t a representation of how much CPU or memory is actively used right now (active usage). Requests are static capacity reservations. Even if our container is idle and only consuming 5MB of memory, Kubernetes still locks the memory slot according to the Requests value (e.g. 512MB) on that Node so it can’t be occupied by other Pods.
2. Resource Limits (The Runtime Phase) #
Limits determine the maximum upper bound of resources a container is allowed to consume during its operational lifetime. When the container starts running, the host-level Linux kernel (through the cgroups or Control Groups mechanism) restricts the container’s power absorption so it doesn’t exceed the Limits boundary.
The system behavior when containers try passing Limits boundaries varies greatly depending on the resource type:
A. CPU (Compressible Resource) #
CPU is categorized as a compressible resource because its processing time allocation can be dynamically compacted. If a container tries consuming CPU beyond its Limits, the Linux kernel doesn’t kill the container.
Instead, the kernel applies CPU Throttling (through CFS Bandwidth Control). The container is forced to wait its turn for processing time (CPU cycles). As a result, our application performance drops drastically, HTTP transaction latency increases, and system responses become slow, but the container keeps running.
B. Memory (Incompressible Resource) #
Memory is categorized as an incompressible resource because data stored in RAM can’t be instantly compacted without damaging process integrity. If a container tries using memory beyond its Limits, the Linux kernel directly takes firm action by killing that container through the Out Of Memory (OOM) Killer mechanism.
The container dies instantly with an exit code of 137 and the Pod status changes to OOMKilled.
The Simple Resource Management Analogy #
To ease developer team understanding, we can use the restaurant table reservation analogy:
- Requests are the number of seats we officially reserve before arriving at the restaurant. The restaurant guarantees those seats will definitely be available when we arrive.
- Limits are the maximum amount of food we’re allowed to order based on our credit card budget.
- CPU Throttling is like a restaurant waiter slowing down food service because we eat too fast beyond our standard package. We’re not kicked out, just have to wait longer.
- OOMKilled is like a restaurant security officer instantly force-kicking us out the restaurant door because we try taking food from other customers’ tables after our maximum budget is exhausted.
Quality of Service (QoS) Classes and Eviction Hierarchy #
Kubernetes automatically groups Pods into one of three Quality of Service (QoS) classes based on the Requests and Limits configuration we write. This QoS class determines the eviction priority when a Node suffers memory crises or runs out of disk space (Node Pressure).
Here’s the pod eviction flow based on QoS classification during Node resource crises:
flowchart TD
NodePressure["1. Resource Pressure Occurs (Node Memory/Disk Pressure)"] --> CheckQoS["2. Scan the QoS Class of All Pods on the Node"]
CheckQoS --> BestEffortList["3. Category: BestEffort (No Requests/Limits)"]
BestEffortList -->|"First Eviction (OOM Score 1000)"| EvictBestEffort["4. Evict & Destroy the BestEffort Pod"]
EvictBestEffort --> CheckPressure{"5. Has the Pressure Subsided?"}
CheckPressure -- "Yes" --> EndEviction["6. Stop Evictions (Cluster Stable)"]
CheckPressure -- "No" --> BurstableList["7. Category: Burstable (Requests < Limits)"]
BurstableList -->|"Second Eviction (OOM Score Based on Usage Percentage)"| EvictBurstable["8. Evict Burstable Pods Consuming Beyond Their Requests"]
EvictBurstable --> CheckPressure2{"9. Has the Pressure Subsided?"}
CheckPressure2 -- "Yes" --> EndEviction
CheckPressure2 -- "No" --> GuaranteedList["10. Category: Guaranteed (Requests == Limits)"]
GuaranteedList -->|"Last Eviction (Minimum OOM Score)"| EvictGuaranteed["11. Forced Eviction of Guaranteed Pods (Critical Condition)"]
EvictGuaranteed --> EndEviction1. Guaranteed QoS (Safest) #
A Pod gets the Guaranteed classification if and only if we set Requests and Limits values with exactly the same numbers for all containers in that Pod (applies to CPU and memory).
# A Pod with Guaranteed QoS
spec:
containers:
- name: api-container
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "500m" # Must equal the requests
memory: "512Mi" # Must equal the requests
- Eviction Priority: Lowest. Kubernetes prioritizes keeping this Pod alive. Guaranteed Pods are only evicted if the Node suffers extreme crisis conditions and all BestEffort and Burstable Pods have been destroyed.
- OOM Score: Rated with a minimum value (usually -997 to 0), making the host Linux kernel reluctant to kill it.
2. Burstable QoS (Medium) #
A Pod is classified as Burstable if we set Limits larger than the Requests values, or if we only specify one of the parameters (e.g. specifying Requests without Limits).
# A Pod with Burstable QoS (commonly used)
spec:
containers:
- name: api-container
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "1000m" # Allowed to burst up to 1 Core when idle
memory: "512Mi" # The memory limit is set conservatively
- Eviction Priority: Medium. If the Node starts getting pressured, Kubernetes sorts Burstable Pods. Burstable Pods consuming memory beyond their Requests capacity are evicted first compared to Burstable Pods still consuming memory below their Requests values.
- OOM Score: Dynamically calculated based on the requested memory percentage against the Node memory capacity.
3. BestEffort QoS (Most Risky) #
Pods enter the BestEffort class if we don’t write any requests and limits configuration at all in all their containers.
# A Pod with BestEffort QoS (Very Dangerous for Production!)
spec:
containers:
- name: debug-tool
# No 'resources' block at all!
- Eviction Priority: Highest. As soon as the Node experiences even the smallest memory pressure, Kubernetes immediately force-kills BestEffort Pods mercilessly to free Node RAM.
- OOM Score: Rated with the maximum number 1000, making it the first easy target for the Linux kernel’s OOM Killer.
Determining Allocation Values Based on Observability Data #
Determining Requests and Limits values must not be done using developer instinct guesses (guesswork). We must use historical application resource usage data recorded in Prometheus.
PromQL Queries for Historical Data Collection #
Use the following PromQL queries on your Prometheus or Grafana dashboards to analyze application characteristics over the last 24 hours or 7 days:
1. Calculating Average CPU Usage (The Basis for Requests Determination) #
This query calculates the container’s average CPU usage to find the daily baseline consumption.
# The average CPU usage of the 'api-service' container in the last 24 hours
avg_over_time(
rate(container_cpu_usage_seconds_total{
namespace="production", container="api-service"
}[5m])[24h:]
)
2. Calculating the 95th Percentile (p95) of CPU Usage (For Limits Determination) #
This query detects CPU consumption values under high workload conditions to anticipate bursts.
# The p95 CPU usage value of the container
quantile_over_time(0.95,
rate(container_cpu_usage_seconds_total{
namespace="production", container="api-service"
}[5m])[24h:]
)
3. Calculating the 99th Percentile (p99) of Memory Usage (For Memory Limits Determination) #
Memory is incompressible, so we must use a high percentile (p99) to avoid OOMKilled during traffic surges.
# The p99 working set bytes memory usage value
quantile_over_time(0.99,
container_memory_working_set_bytes{
namespace="production", container="api-service"
}[24h:]
)
The Practical Initial Tuning Formula (Rule of Thumb) #
Based on the metrics above, use the following practical formula to determine initial allocations before releasing applications to production:
$$\text{CPU Requests} = \text{Daily average CPU} \times 1.20 \quad (\text{20% Buffer})$$ $$\text{Memory Requests} = \text{Daily average Memory} \times 1.30 \quad (\text{30% Buffer})$$ $$\text{Memory Limits} = \text{Daily p99 Memory} \times 1.50 \quad (\text{50% Safety Buffer})$$
[!TIP] CPU Limits Tuning Recommendation: At the production level, consider not setting CPU Limits (leave them empty) if your application is sensitive to transaction latency and you’ve already applied limits through ResourceQuotas at the namespace level. CPU Throttling from overly tight limits is often the mysterious cause of slow Java/Go application responses, even though the physical Node CPU capacity is still very loose.
LimitRange: Default Protection at the Namespace Level #
To prevent developers accidentally deploying containers without resource specs (thus entering BestEffort QoS), we must apply LimitRange policies in every active namespace.
LimitRanges automatically insert default Requests and Limits values if the submitted Pod manifests don’t have them.
# File: k8s/namespaces/production-limitrange.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: prod-default-limits
namespace: prod-apps
spec:
limits:
- type: Container
# Automatically insert Limits if developers leave them empty
default:
cpu: "500m"
memory: "512Mi"
# Automatically insert Requests if left empty
defaultRequest:
cpu: "200m"
memory: "256Mi"
# Limit the maximum values developers may declare
max:
cpu: "4000m"
memory: "8Gi"
# Limit the minimum values developers may declare
min:
cpu: "50m"
memory: "64Mi"
- type: PersistentVolumeClaim
min:
storage: "1Gi"
max:
storage: "100Gi"
If developers try deploying Pods requesting memory: 16Gi (exceeding the LimitRange max limit of 8Gi), the Kubernetes API Server immediately rejects the deployment process with a validation error message.
ResourceQuota: Team Total Capacity Restrictions #
If our cluster is shared by several teams (multi-tenant clusters), one team can accidentally deploy dozens of high-Requests Pods consuming the cluster’s entire physical capacity, leaving zero capacity for other teams.
ResourceQuotas limit the accumulated amount of resources consumable in one namespace.
# File: k8s/namespaces/team-a-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-a-resource-limits
namespace: team-a-dev
spec:
hard:
# CPU & Memory Requests Accumulation Limits
requests.cpu: "10" # Maximum 10 Cores of total CPU requests
requests.memory: "20Gi" # Maximum 20 GiB of total RAM requests
# CPU & Memory Limits Accumulation Limits
limits.cpu: "20"
limits.memory: "40Gi"
# Physical Object Count Limits in the Namespace
pods: "30" # Maximum of only 30 active Pods
services: "10" # Maximum 10 Services
persistentvolumeclaims: "5" # Maximum 5 PVCs
# Total Storage Limits
requests.storage: "100Gi"
We can check the remaining quota usable by the team using the following command:
# Show the active quota limit status and used capacity
kubectl describe resourcequota team-a-resource-limits -n team-a-dev
The description output presents a detailed comparison:
Resource Used Hard
-------- --- ----
limits.cpu 4500m 20
limits.memory 9Gi 40Gi
pods 12 30
requests.cpu 2200m 10
requests.memory 4.5Gi 20Gi
Vertical Pod Autoscaler (VPA) for Automation #
Manually managing requests/limits values for hundreds of microservice applications is very tiring. We can leverage the Vertical Pod Autoscaler (VPA) to continuously analyze actual container usage and recommend or automatically apply allocation adjustments.
# File: k8s/production-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: payment-api-vpa
namespace: prod-apps
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: payment-api
updatePolicy:
# Mode Options:
# - "Off": Only gives recommendations in the VPA status, doesn't change Pods (Highly Recommended for Production)
# - "Auto": Automatically evicts old Pods and creates new ones with the latest request/limits (Causes Pod restarts)
updateMode: "Off"
resourcePolicy:
containerPolicies:
- containerName: app-container
minAllowed:
cpu: "100m"
memory: "128Mi"
maxAllowed:
cpu: "2000m"
memory: "4Gi"
To view the container size recommendations generated by the VPA:
# Read the container adjustment recommendation metrics
kubectl describe vpa payment-api-vpa -n prod-apps
The status output shows the target recommendations:
Recommendation:
Container Recommendations:
Container Name: app-container
Lower Bound:
Cpu: 150m
Memory: 256Mi
Target: # The ideal request values from VPA analysis
Cpu: 280m
Memory: 380Mi
Upper Bound:
Cpu: 800m
Memory: 768Mi
Resource Management Anti-Patterns #
Avoid the following two fatal mistakes when designing resource allocations in production clusters:
1. Letting Applications Run Without Limits (BestEffort Class) #
# ANTI-PATTERN: A Deployment without any resource declarations
apiVersion: apps/v1
kind: Deployment
metadata:
name: untracked-api
spec:
template:
spec:
containers:
- name: app
image: app:latest # DON'T: The Pod enters BestEffort QoS!
Operational Risks:
- BestEffort Pods are instantly force-killed by the kernel (OOMKilled) when the Node experiences even the smallest RAM consumption increase.
- The absence of requests makes the scheduler blind; Pods can be scheduled to Nodes whose physical RAM is already exhausted by the host OS, triggering Node system failures.
✓ SOLUTION: Install LimitRanges in namespaces and make sure CI/CD pipelines validate the existence of resources parameters in every YAML manifest.
2. Overcommitting Memory with Too-Wide Limit Gaps #
Configuring very small memory Requests values but setting very large memory Limits to avoid OOMKilled errors without mature calculations.
# ANTI-PATTERN: Request and Limit memory gaps that are too far apart
resources:
requests:
cpu: "100m"
memory: "64Mi" # Very Small
limits:
cpu: "1000m"
memory: "16Gi" # Very Large (Dangerous overcommit)
Operational Risks:
- The Kubernetes Scheduler schedules this Pod to a Node with only 512MB of remaining Allocatable RAM (because the Pod only requests 64MB).
- When the application runs and suffers workload surges, it requests up to 4GB of memory allocation.
- Because its limit is 16GB, the container feels allowed to use that RAM. However, the physical Node capacity only has 512MB left.
- As a result, the physical Node suffers total memory exhaustion (Out-Of-Memory). The host Linux kernel is forced to kill critical OS processes, making the Node 'NotReady' and killing all other neighboring Pods.
✓ SOLUTION: Keep the memory requests vs limits ratio no more than 1:2 for production environments, or use Guaranteed QoS (1:1) for full stability.
Resource Management Audit Checklist #
Do a comprehensive audit on your production cluster using the following checklist guide:
REQUESTS & LIMITS CONFIGURATION:
□ No Pods in the production namespace run with the 'BestEffort' QoS Class (no resource spec).
□ Requests values are determined based on real historical usage data (not guesses/instincts).
□ The Memory requests vs limits allocation ratio is within safe limits (maximum 1:2 ratio for Burstable).
□ Critical workloads like main databases are configured using the 'Guaranteed' QoS Class.
□ CPU Limits are carefully configured (or left empty) to avoid latency from CPU Throttling.
NAMESPACE & DEFAULT PROTECTION:
□ Every active namespace has a 'LimitRange' object to insert default CPU/RAM values.
□ 'ResourceQuota' objects are installed in every multi-tenant namespace to limit accumulated CPU/RAM consumption.
□ ResourceQuota policies include critical object count limits (Pods, PVCs, Services).
□ Storage quotas are specifically configured based on StorageClasses.
MONITORING & AUTOSCALING:
□ PromQL queries for p95 CPU and p99 Memory are periodically monitored through Grafana dashboards.
□ The Vertical Pod Autoscaler (VPA) is installed in 'Off' mode to provide daily right-sizing recommendations.
□ Alarm notifications are active to detect containers approaching memory limit boundaries (>90% memory usage).
□ Load testing is run to validate container CPU throttling performance.
Summary #
- Requests for Scheduling, Limits for Runtime — Understand the difference; the scheduler uses requests to choose Nodes, while the host kernel uses limits to physically restrict container consumption.
- Beware of OOMKilled Impacts — Remember that exceeding memory limits triggers instant container death (OOMKilled - exit code 137), while exceeding CPU limits only triggers performance degradation (throttling).
- Apply Guaranteed QoS for DBs — Configure requests and limits with identical values for important database pods to get the lowest eviction priority from Kubernetes.
- Use LimitRanges as Safety Nets — Install LimitRange objects at the namespace level to secure clusters from developer negligence forgetting to write container resource specs.
- Prevent Monopolies with ResourceQuotas — Limit accumulated CPU/RAM resources per team at the namespace level using ResourceQuotas for resource allocation fairness.
- Avoid Extreme Memory Overcommit — Keep the memory requests vs limits ratio rational to protect physical Nodes from total memory starvation threats.
← Previous: Ecosystem & Tooling Anti-Patterns Next: Autoscaling →