Health Check #
One of Kubernetes’s most powerful features is its ability to independently detect application failures and recover automatically (self-healing), as well as dynamically route network traffic (service routing). However, for Kubernetes to do these actions accurately, it must know the actual health condition of running application containers. Without proper Health Check configuration, the Kubernetes API Server only assesses Pods based on the container lifecycle status reported by the container runtime. Containers suffering deadlocks, internal memory exhaustion, or disconnection from the main database are still considered running normally (Running) by Kubernetes, even though those applications can no longer serve users. This article deeply dissects the three health check types (probes) in Kubernetes along with their optimal configuration parameters.
Probe Lifecycle and Evaluation Flow #
The Kubernetes Kubelet on every node actively evaluates container status using three probe types. The diagram below shows container state transitions from being scheduled to serving traffic and being continuously evaluated in the background.
flowchart TD
StartPod["1. Pod Starts / Container Runs"] -->|"Start Evaluation"| StartupLoop{"2. Is a startupProbe configured?"}
StartupLoop -- "Yes" --> ExecStartup{"3. Run the startupProbe"}
ExecStartup -- "Failed" --> CheckStartupLimit{"4. Is the failureThreshold exceeded?"}
CheckStartupLimit -- "Yes" --> RestartStartup["5. Container Restarted by the Kubelet"]
CheckStartupLimit -- "No" --> WaitStartup["Wait periodSeconds, Test Again"] --> ExecStartup
StartupLoop -- "No / Success" --> ActiveState["6. Container Active (Liveness & Readiness Start)"]
ExecStartup -- "Success" --> ActiveState
subgraph LivenessLoop["Liveness Cycle (Background)"]
direction TB
ExecLive{"Run the livenessProbe"}
ExecLive -- "Fails N Times" --> RestartLive["Restart the Container"]
ExecLive -- "Success" --> WaitLive["Wait periodSeconds"] --> ExecLive
end
subgraph ReadinessLoop["Readiness Cycle (Background)"]
direction TB
ExecReady{"Run the readinessProbe"}
ExecReady -- "Fails N Times" --> RemoveEndpoints["Remove the Pod IP from the Service Endpoints"]
ExecReady -- "Success" --> AddEndpoints["Add the Pod IP to the Service Endpoints"]
RemoveEndpoints --> WaitReady["Wait periodSeconds"] --> ExecReady
AddEndpoints --> WaitReady
end
ActiveState --> LivenessLoop
ActiveState --> ReadinessLoopThe Three Probe Types in Kubernetes #
Kubernetes provides three probe types with fundamentally different operational purposes and failure consequences:
The Three Main Probe Types in Kubernetes:
1. Startup Probe:
- Purpose: Know when the application finishes its startup and initial initialization process.
- Behavior: While the startupProbe runs and hasn't succeeded, the livenessProbe and readinessProbe are disabled.
- Failure Consequence: The Kubelet kills the container and restarts it.
- Use Case: Java applications slow at initial initialization, or applications doing large data caching at startup.
2. Liveness Probe:
- Purpose: Know whether the application container is still alive and running normally internally.
- Behavior: Starts running after the startupProbe succeeds (or runs immediately if the startupProbe is omitted).
- Failure Consequence: The Kubelet kills the container and restarts it to recover the state.
- Use Case: Detecting process deadlocks, infinite loops, or critical memory leaks.
3. Readiness Probe:
- Purpose: Know whether the application is ready to receive and serve network traffic.
- Behavior: Continuously evaluated while the pod runs.
- Failure Consequence: The Pod IP is removed from the Service Endpoints (the container is NOT restarted). Traffic is redirected to other pods.
- Use Case: Detecting temporary database connection failures, cutting traffic when pods do graceful shutdowns.
The Danger of Cascade Failures (Liveness Probe Cascade Failure) #
Understanding the difference between Liveness and Readiness is crucial. One of the most commonly encountered architectural disasters in production environments is the Liveness Probe Cascade Failure.
Disaster Scenario: The Database Experiences a Temporary Outage
[WITH a Liveness Probe Checking the Database (WRONG)]
1. The database is overloaded/down for 30 seconds.
2. Liveness probes on all application Pods detect database connection failures.
3. The Kubelet detects liveness failures on ALL application Pods simultaneously.
4. The Kubelet restarts all application Pods at the same time.
5. When the new applications wake up, they re-initialize and bombard the just-recovered database with new connections.
6. The database suffers overload again due to the startup connection surge.
7. The cluster is stuck in a mass restart cycle (CrashLoopBackOff).
[WITH a Readiness Probe Checking the Database (CORRECT)]
1. The database is overloaded/down for 30 seconds.
2. Readiness probes on all application Pods detect database connection failures.
3. The Kubelet removes all Pod IPs from the Service Endpoints.
4. User traffic is temporarily stopped (Inbound Traffic Blocked) or directed to a static fallback page.
5. The application containers are NOT restarted (stay alive).
6. When the database recovers, readiness probes detect an OK connection.
7. Pods are automatically added back to the Service Endpoints.
8. The system recovers smoothly without a CrashLoopBackOff cycle.
The Golden Rule: Never check external dependencies (like databases, Redis caches, or third-party APIs) inside Liveness Probe queries. Liveness probes may only assess the container’s own internal health.
Probe Handler Types #
Kubernetes supports four mechanisms (handlers) for executing health checks:
1. HTTP GET Request #
The Kubelet sends an HTTP GET request to the Pod IP on a specific port and path. Responses with status codes >= 200 and < 400 are considered successful.
livenessProbe:
httpGet:
path: /health/live
port: 8080
httpHeaders:
- name: X-Custom-Header
value: kubelet-liveness
2. TCP Socket Connection #
The Kubelet tries opening a TCP connection to the container port. If the port is open, the probe is declared successful. Very useful for non-HTTP services like databases (PostgreSQL/MySQL) or Redis.
readinessProbe:
tcpSocket:
port: 5432
3. gRPC Health Check #
Starting from Kubernetes v1.24, the Kubelet supports native gRPC health checks following the gRPC Health Checking Protocol.
livenessProbe:
grpc:
port: 50051
service: billing-service
4. Exec Command #
The Kubelet executes a command inside the container process space. The command is declared successful if it returns an exit code of 0.
readinessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
Writing Proper Health Check Endpoints (Multi-Language) #
For health checks to work optimally, we must write separate /health/live and /health/ready endpoint routes inside our application code.
Here’s a comparison between wrong endpoint logic writing (one combined endpoint) and correct, safe endpoint implementation in FastAPI (Python) and Express (Node.js).
// ANTI-PATTERN (Node.js/Express): One '/health' endpoint for liveness and readiness,
// and executing heavy database queries on every call.
// This burdens the database CPU because the Kubelet calls this endpoint every 5 seconds.
app.get('/health', async (req, res) => {
const dbOk = await checkDatabaseConnection();
if (!dbOk) {
return res.status(500).send("Database Down"); // DON'T: Liveness failure triggers a restart loop
}
res.status(200).send("OK");
});
// ==============================================================================
// CORRECT: Separating Liveness and Readiness with Isolated Logic.
// File: server.js
const express = require('express');
const app = express();
let isGracefulShutdown = false;
// Catch the termination signal from Kubernetes to start shutdown mode
process.on('SIGTERM', () => {
isGracefulShutdown = true;
});
// 1. Liveness Endpoint: Only check internal health (Thread/Memory)
app.get('/health/live', (req, res) => {
// Check if an internal deadlock or memory consumption beyond the critical limit occurs
if (isAppDeadlocked()) {
return res.status(500).send("Internal Deadlock Detected");
}
res.status(200).send("Alive");
});
// 2. Readiness Endpoint: Check external connection availability and shutdown mode
app.get('/health/ready', async (req, res) => {
// If the pod is in the termination process (graceful shutdown), turn off readiness
// so the Kubernetes Ingress/Service stops sending new requests.
if (isGracefulShutdown) {
return res.status(503).send("Shutting Down");
}
try {
// Do a lightweight database connection check (SELECT 1)
await db.raw('SELECT 1');
res.status(200).send("Ready");
} catch (error) {
res.status(503).send("Database Connection Lost");
}
});
Probe Timing Configuration Parameters #
We must set the check timing parameters precisely to avoid slow detection or premature restarts.
# Example of an Optimal Probe Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-deployment
namespace: production
spec:
replicas: 3
template:
spec:
containers:
- name: app
image: billing:v1.2.0
ports:
- containerPort: 8080
name: http-port
# 1. Startup Probe: Give a startup time tolerance of up to 5 minutes
startupProbe:
httpGet:
path: /health/live
port: http-port
failureThreshold: 30 # Try up to 30 times
periodSeconds: 10 # Evaluate every 10 seconds (30 x 10s = 300 seconds/5 minutes)
timeoutSeconds: 5 # The response timeout limit
# 2. Liveness Probe: Starts after startup succeeds
livenessProbe:
httpGet:
path: /health/live
port: http-port
periodSeconds: 15 # Check every 15 seconds
failureThreshold: 3 # Restart after 3 consecutive failures (45 seconds)
timeoutSeconds: 5
successThreshold: 1 # 1 success is enough to return to the healthy status
# 3. Readiness Probe: Continuously evaluated for service routing
readinessProbe:
httpGet:
path: /health/ready
port: http-port
periodSeconds: 5 # Check more often (5 seconds) for fast responses
failureThreshold: 3 # Remove from the Service if it fails 3 times (15 seconds)
timeoutSeconds: 3
successThreshold: 1 # Immediately add back to the Service on 1 success
Parameter Relationships: #
initialDelaySeconds: Since thestartupProbeexists, this parameter must be set to0(or omitted) for liveness and readiness, becausestartupProbeacts as an automatic, dynamic evaluation delay.successThreshold: This value must be set to1for liveness probes (cannot be changed), to ensure status recovery happens instantly.
Health Check Management Anti-Patterns #
Here are critical configuration mistakes often found in Kubernetes clusters:
1. Connecting Liveness Probes to Third-Party APIs #
If our application uses a third-party payment gateway (e.g. Stripe/Midtrans) and their API suffers downtime, connecting the liveness probe to that API makes our application container restart endlessly. This is a fatal logic mistake.
2. Setting failureThreshold Too Small (Value 1) #
Setting failureThreshold: 1 on readiness probes. If a 100-millisecond internal cluster network fluctuation (network hiccup) occurs, the Pod is immediately removed from the Service, triggering unnecessary orchestrator system panic. Use a safe minimum limit of 3.
3. Not Having Resource Limits on Health Check Endpoints #
Writing health check logic executing heavy mathematical calculations or complex encryption on the /health endpoint. This endpoint is constantly called by the Kubelet. If the endpoint consumes lots of CPU resources, the health check process itself can cause main application performance degradation.
Production Health Check Audit Checklist #
Make sure all your workload Pods meet the following checklist standards before being deployed to production:
CODE ENDPOINT DESIGN:
□ The '/health/live' and '/health/ready' endpoints are created separately in the code.
□ The '/health/live' endpoint only tests internal health (memory, deadlock).
□ The '/health/ready' endpoint tests the main database connection and cache startup status.
□ The '/health/ready' endpoint detects SIGTERM signals to turn off the ready status during graceful shutdowns.
□ Endpoint logic is designed as concisely as possible to not burden container CPU/Memory usage.
KUBERNETES MANIFEST CONFIGURATION:
□ Every container in the Pod is configured with a livenessProbe and readinessProbe.
□ Applications with initial startup times > 30 seconds use a 'startupProbe'.
□ The 'initialDelaySeconds' parameter is set to '0' if a 'startupProbe' is enabled.
□ The 'failureThreshold' parameter is set to a minimum of '3' to filter false positives.
□ The probe handler type used matches the application characteristics (HTTP, TCP, gRPC, Exec).
□ The 'timeoutSeconds' parameter is configured lower than 'periodSeconds'.
Summary #
- Liveness Evaluates Internals, Readiness Evaluates Readiness — Understand the functional difference; liveness triggers container restarts, while readiness routes Service traffic.
- Don’t Check Databases in Liveness Probes — Connecting liveness probes to databases triggers cascade failure disasters when the database goes down; use readiness probes to check database connections.
- startupProbe Replaces Static Delays — Use startupProbes to accommodate slow application startups instead of statically extending liveness delay parameters.
- Enable Graceful Shutdown Through Readiness — Make sure the readiness endpoint listens for SIGTERM signals to remove pods from the Service before the application is shut down.
- Set failureThreshold to a Minimum of 3 — Avoid setting the failure tolerance limit to 1 to prevent pod removal from traffic due to momentary network disruptions.
- Choose the Right Handler — Leverage native gRPC handlers for gRPC-based microservice applications for check accuracy without HTTP/Exec bridges.
← Previous: Grafana Dashboard Next: Observability Anti-Patterns →