Autoscaling #

One of the main promises of adopting cloud-native architecture is dynamic resource elasticity. In traditional environments, we must design infrastructure based on peak traffic capacity, which means dozens of servers sit idle with no activity at night, wasting company budgets. In Kubernetes, we can automate compute capacity adjustments in real-time based on workload fluctuations. However, configuring poorly planned autoscaling can be very dangerous: Pods scaling too late can cause service outages, policy collisions between horizontal and vertical scaling can trigger scale wars, and unintegrated cluster autoscalers can cause new Pods to get stuck in the Pending status forever. Autoscaling in Kubernetes is a multi-dimensional system. This article comprehensively discusses the three main autoscaling dimensions (HPA, VPA, Cluster Autoscaler), event-based scaling mechanisms using KEDA, operational failure mitigations, and synergistic integration at the production level.

The Three Autoscaling Dimensions #

To build a truly elastic system, we must understand the three autoscaling dimensions in Kubernetes and how they interact across layers:

flowchart TD
    subgraph PodLevel["Pod Layer (Workload Scaling)"]
        direction LR
        HPA["HPA (Horizontal Scaling: Add/Remove Pods)"]
        VPA["VPA (Vertical Scaling: Resize Pod CPU/RAM)"]
    end
    
    subgraph NodeLevel["Node Layer (Server Scaling)"]
        CA["Cluster Autoscaler (Add/Remove Physical Nodes)"]
    end
    
    Traffic["Traffic / Workload Fluctuations"] --> HPA
    Traffic --> VPA
    
    HPA -. "Pods Can't Be Scheduled (Pending)" .-> CA
    VPA -. "Nodes Run Out of Compute Space" .-> CA
  1. Horizontal Pod Autoscaler (HPA): Dynamically adds or reduces Pod replica counts. “More instances to divide the load.”
  2. Vertical Pod Autoscaler (VPA): Changes the CPU and memory allocation capacity (Requests/Limits) of existing Pods. “Give the Pod a properly fitting shirt size.”
  3. Cluster Autoscaler (CA): Adds or reduces the number of physical/virtual Nodes at the cloud provider level. “Add new servers if the house for Pods is full.”

Horizontal Pod Autoscaler (HPA) #

HPA works by monitoring container usage metrics through the Metrics Server API intermediary. The kube-controller-manager runs periodic HPA evaluation cycles (default every 15 seconds).

The HPA Scaling Algorithm Formula #

HPA calculates the target replica count using the following official mathematical formula:

$$\text{TargetReplicas} = \left\lceil \text{CurrentReplicas} \times \left( \frac{\text{CurrentMetricValue}}{\text{DesiredMetricValue}} \right) \right\rceil$$

Production Case Example: #

  • Our application currently runs with 3 replicas (CurrentReplicas = 3).
  • The CPU usage target we want is 50% (DesiredMetricValue = 50).
  • Prometheus reports the current average active CPU usage across all pods is 80% (CurrentMetricValue = 80).

$$\text{TargetReplicas} = \left\lceil 3 \times \left( \frac{80}{50} \right) \right\rceil = \lceil 3 \times 1.6 \rceil = \lceil 4.8 \rceil = 5 \text{ replicas}$$

Kubernetes immediately sends a command to the ReplicaSet to raise the pod count to 5 replicas instantly.

Comprehensive HPA Manifest (autoscaling/v2 Syntax) #

Since the v2 version release, HPA supports monitoring multiple metrics simultaneously (e.g. watching CPU, Memory, and HTTP RPS metrics at once) plus scaling behavior adjustments (behavior policies).

# File: k8s/production-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: billing-api-hpa
  namespace: prod-apps
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: billing-api
  minReplicas: 3 # Never set to 1 in production for HA
  maxReplicas: 30
  metrics:
  # 1. Internal metric: CPU Utilization
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65 # The CPU utilization target of 65%
  
  # 2. Internal metric: Memory (Use the absolute AverageValue, not percentages)
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: "350Mi"
  
  # 3. Custom Metric: HTTP Requests Per Second (RPS) via the Prometheus Adapter
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "800" # Target 800 req/s per Pod
        
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0 # Immediately scale up without waiting if load surges
      policies:
      - type: Percent
        value: 100 # Allow doubling the pod count (100% growth)
        periodSeconds: 15
      - type: Pods
        value: 4 # Or add a minimum of 4 new pods
        periodSeconds: 15
      selectPolicy: Max # Choose the policy producing the most pods
      
    scaleDown:
      # The stabilization window holds HPA back from immediately reducing pods during momentary traffic drops
      stabilizationWindowSeconds: 300 # Wait 5 minutes to prevent flapping/oscillation
      policies:
      - type: Percent
        value: 10 # Reduce a maximum of 10% of active replicas every 60 seconds
        periodSeconds: 60

Metrics Server: The Heart of HPA Observability #

HPA can’t function without a Metrics Server in the cluster. The Metrics Server acts as a lightweight data aggregator collecting CPU and memory usage metrics from the kubelet agent on every Node (through the /stats/summary endpoint), then presenting them standardized to the Kubernetes API Server via the metrics.k8s.io extension.

# 1. Install the latest Metrics Server version directly from the official repository
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 2. Verify the resource monitoring health status
kubectl top nodes
kubectl top pods -A

[!NOTE] Configuration for Local Clusters (Minikube/Kind): By default, the Metrics Server fails to run in local clusters because Kubelet certificates aren’t signed by the cluster’s official CA. We must edit the Metrics Server Deployment object and add the --kubelet-insecure-tls flag to its container argument parameters so it ignores SSL certificate validation.

KEDA: Kubernetes Event-Driven Autoscaling #

The built-in Kubernetes HPA is very reliable when monitoring internal metrics like CPU and memory usage. However, for worker queue type applications, CPU metrics are often misleading.

A worker pod processing heavy task queues might consume 99% CPU to process a single message. Conversely, when 10,000 messages pile up in RabbitMQ but the worker is stuck waiting for a database connection, the worker’s CPU usage is actually at 1%. The standard HPA detects that 1% CPU and leaves the pod count low, causing the queue to keep snaking longer.

KEDA solves this problem by acting as a bridge presenting metrics from dozens of external systems (RabbitMQ, Apache Kafka, Redis, PostgreSQL, AWS SQS) to the HPA API Server natively.

flowchart LR
    ExternalSystem["External Systems (RabbitMQ / Kafka Queue)"] <-->|"1. Pull Metrics"| KEDAOpertor["KEDA Operator Controller"]
    KEDAOpertor -->|"2. Present Metrics via"| MetricsAdapter["KEDA Metrics Adapter (v1alpha1)"]
    MetricsAdapter <-->|"3. Query Targets"| K8sHPA["Standard Kubernetes HPA"]
    K8sHPA -->|"4. Scale Replicas"| Deployment["Worker Deployment Pods"]

1. RabbitMQ Queue Length-Based Scaling (ScaledObject) #

Here’s a KEDA manifest file for dynamically scaling worker Pods based on the message count in RabbitMQ:

# File: k8s/keda-rabbitmq-scaler.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-rabbitmq-scaler
  namespace: prod-apps
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: task-worker
  minReplicaCount: 0 # KEDA supports scale-to-zero to save costs!
  maxReplicaCount: 50
  cooldownPeriod: 300 # Holds replicas for 5 minutes after the queue empties
  triggers:
  - type: rabbitmq
    metadata:
      host: amqp://guest:***@rabbitmq-service.prod-apps.svc.cluster.local:5672
      queueName: payment-tasks
      queueLength: "15" # Add 1 new Pod for every 15 messages in the queue

2. Time-Based Scaling (ScaledCronJob) #

For workloads with very predictable time-based patterns (e.g. an employee attendance app always busy at 8 AM and quiet at night), we can use KEDA’s ScaledCronJob:

# File: k8s/keda-cron-scaler.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: office-hours-scaler
  namespace: prod-apps
spec:
  scaleTargetRef:
    name: employee-portal
  minReplicaCount: 2
  maxReplicaCount: 15
  triggers:
  - type: cron
    metadata:
      timezone: "Asia/Jakarta"
      start: "0 7 * * 1-5"       # 07:00 AM WIB on Monday-Friday
      end: "0 18 * * 1-5"         # 06:00 PM WIB
      desiredReplicas: "10"      # Force raise to a minimum of 10 replicas during working hours

Vertical Pod Autoscaler (VPA) #

When HPA scales by quantity (adding pods), the VPA scales by quality (changing Pod CPU/Memory request sizes). The VPA consists of three main components running as a controller:

  1. Recommender: Continuously monitors actual container usage and calculates optimal target allocation recommendations.
  2. Updater: Monitors active Pods. If active Pods have configurations not matching the Recommender’s recommendations, the Updater evicts those Pods to be redeployed.
  3. Admission Controller: A webhook intercepting new Pod creation requests from the Kubernetes API Server, then replacing the Kubelet resource spec values according to the latest recommendations.
# File: k8s/production-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: backend-vpa
  namespace: prod-apps
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend-service
  updatePolicy:
    # updateMode "Off" is recommended for production. 
    # The VPA only calculates recommendations and doesn't force-restart active Pods.
    # We can safely apply the recommendations in the next CI/CD cycle.
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: app-container
      minAllowed:
        cpu: "100m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2000m"
        memory: "4Gi"

Cluster Autoscaler (CA) #

The Cluster Autoscaler operates at the physical Node infrastructure level. It monitors the cluster for scheduling failure signs.

Scaling Trigger Conditions #

1. Scale-Up Conditions (Adding Nodes) #

The CA continuously scans the cluster to detect whether any Pods are in the Pending status with specific reasons: CPU capacity shortages, Memory shortages, or Node selector mismatches. If these conditions are met, the CA communicates with the Cloud Provider API (AWS Auto Scaling Groups, GCP Instance Groups, Azure VMSS) to launch new VMs and add them to the cluster as additional Nodes.

2. Scale-Down Conditions (Reducing Nodes) #

A Node is considered a deletion candidate if:

  • The total Requests of all Pods running on that Node are below the utilization threshold (default 50%).
  • All Pods running on that Node can be moved to other existing Nodes without triggering new Pending statuses.
  • The Node has been idle for at least 10 minutes.

[!CAUTION] Protecting Critical Pods from Scale-Down Evictions: By default, the Cluster Autoscaler is allowed to move Pods to empty Nodes for cost savings. If we have critical Pods (like transaction processing pods that must not be disturbed mid-way), we must attach a safety lock annotation so the Node it runs on is never deleted by the Cluster Autoscaler.

# Add at the Deployment Pod Template metadata level
metadata:
  annotations:
    # Prevents the Cluster Autoscaler from killing this Node for cost efficiency
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Policy Collisions: HPA vs VPA (Scale Wars) #

One of the most fatal infrastructure design mistakes is configuring HPA and VPA simultaneously to monitor the same metrics (e.g. CPU) on the same target Deployment. This triggers the Scale War condition:

HPA and VPA Conflict Scenario:
  1. Traffic load rises -> Actual CPU usage on Pods exceeds the limits.
  2. The VPA detects the high load -> The VPA decides to raise the Pod CPU Request capacity (Vertically).
  3. To apply the new CPU, the VPA (Auto mode) kills the Pod for a restart.
  4. The active Pod count suddenly drops in the cluster.
  5. The HPA detects the reduced active pod count and high transaction load.
  6. The HPA decides to add Pod replica counts (Horizontally).
  7. When new replica Pods start, the workload is evenly divided across all pods.
  8. The VPA detects the average CPU usage per pod decreasing because replicas increased.
  9. The VPA decides to reduce the Pod CPU Request capacity.
  10. This process keeps spinning without ever reaching a stable point (flapping loop).

Correct Production Integration Guidelines #

To avoid the collision above, apply the following design rules:

  • Rule 1 (Metric Separation): If you want to actively use both, configure the HPA to monitor CPU (scaling based on transaction load) and configure the VPA to only monitor Memory (vertically adjusting RAM capacity so it doesn’t suffer OOMKilled).
  • Rule 2 (Non-Intrusive Recommendations): Set updateMode: "Off" on the VPA spec. This way, the VPA only acts as a smart advisor (Recommender) writing allocation suggestions in the manifest status. Developer teams can read those recommendations on schedule and update Helm values.yaml files in GitOps pipelines without unexpected automatic restarts.

Autoscaling Practice Anti-Patterns #

Avoid the following two configuration mistakes in your production cluster:

1. Setting HPA minReplicas: 1 in Production Environments #

Setting the minimum HPA pod replica count to 1 to save compute budgets.

# File: k8s/production-hpa.yaml
# ANTI-PATTERN: Setting minReplicas too low in production
spec:
  minReplicas: 1 # DON'T!
  maxReplicas: 10

Operational Risks:
- No High Availability: When that single Pod dies suddenly (e.g. due to node network disruptions or OOMKilled), the application suffers total service downtime for dozens of seconds until Kubernetes finishes detecting the failure and recreates the replacement pod.
- Cold-Start Delays: During sudden traffic surges, that single pod gets overwhelmed and crashes before the HPA manages to trigger the second pod creation.
✓ SOLUTION: Always set 'minReplicas' to a minimum of 2 (3 recommended) in production so traffic can be evenly divided across physical availability zones.

2. Too-Short Stabilization Windows (Thrashing / Flapping) #

Setting the stabilizationWindowSeconds parameter on scaleDown policies with too-short durations (e.g. less than 60 seconds).

# ANTI-PATTERN: Too-aggressive scaleDown policies
behavior:
  scaleDown:
    stabilizationWindowSeconds: 10 # DON'T: Too sensitive!

Operational Risks:
- The cluster suffers 'flapping' (or thrashing) conditions. When a short 10-second transaction surge happens, the HPA triggers node/pod scale-ups. Ten seconds later when transactions decline, the HPA immediately kills those pods. This repeats endlessly.
- Container booting processes (especially slow-starting Java Spring Boot) consume lots of cluster CPU resources just for futile initialization processes because pods are destroyed right after starting.
✓ SOLUTION: Use the default scaleDown stabilization value of at least 300 seconds (5 minutes) to give sufficient traffic waiting time before deciding to reduce instances.

Autoscaling Configuration Audit Checklist #

Use this checklist to verify your production cluster’s elasticity functionality:

HORIZONTAL POD AUTOSCALER (HPA):
  □ The minimum replica count ('minReplicas') across all production HPAs is set to at least 2 (or 3).
  □ The 'behavior.scaleDown.stabilizationWindowSeconds' parameter is configured for at least 300 seconds.
  □ CPU utilization targets aren't set too high (recommended safe limit 60-70% to leave burst room).
  □ Memory metrics aren't used as the sole HPA trigger (unless combined with CPU/RPS metrics).
  □ The Metrics Server is installed healthily and reports data without SSL certificate obstacles.

KEDA & EVENT-DRIVEN SCALING:
  □ KEDA is used to scale worker Pods based on real queue metrics (not CPU).
  □ Credential connections to message brokers (like RabbitMQ/Kafka) are secured using TriggerAuthentication.
  □ The scale-to-zero feature ('minReplicaCount: 0') has its cold-start path stability tested.
  □ ScaledCronJobs are configured using the correct local time zones ('timezone: Asia/Jakarta').

VERTICAL POD AUTOSCALER & CLUSTER AUTOSCALER:
  □ HPA and VPA aren't configured to monitor the same metrics on the same Deployment.
  □ The VPA is set using 'Off' (recommendation) or 'Initial' mode in production environments.
  □ The Cluster Autoscaler is integrated with cloud provider ASGs/Instance Groups with rational max-node limits.
  □ Critical system Pods (like CoreDNS, Ingress Controllers) have the 'safe-to-evict: false' annotation.

Summary #

  • Understand the Three Dimensions — Master the role separation of HPA (adding pods), VPA (resizing pods), and the Cluster Autoscaler (adding physical nodes) so elastic systems run harmoniously.
  • Prevent Flapping with Stabilization — Always set stabilizationWindowSeconds to at least 5 minutes on HPA scaleDown rules to dampen scaling oscillations from momentary traffic surges.
  • HA Requires a Minimum of 2 Replicas — Never set minReplicas: 1 on production clusters; single pod failures trigger instant service outages for users.
  • Use KEDA for Queues — Leave CPU monitoring behind for worker-type pods; use KEDA to measure real queue lengths in RabbitMQ or Kafka.
  • Avoid HPA-VPA Collisions — Keep allocation disputes far away by strictly separating HPA monitoring metrics (CPU/RPS) and VPA (Memory only).
  • Secure Critical Pods from Evictions — Attach the safe-to-evict: false annotation to main system pods so the Cluster Autoscaler doesn’t kill the node they run on for cost efficiency.

← Previous: Resource Management   Next: High Availability →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact