Infrastructure Contract #

Kubernetes is more than just a container orchestration tool. Its existence redefines how developer teams and platform/operations teams (DevOps/SRE) collaborate. At the center of this modern collaboration sits a fundamental concept that is crucial yet rarely discussed in depth: the Infrastructure Contract.

An infrastructure contract is a two-way agreement — both implicit and explicit — between the application (container) and the platform (Kubernetes). This agreement determines what the platform must provide to guarantee container stability, and conversely, what the application must promise so the platform can make autonomous, smart decisions when incidents happen.

Understanding the Two-Way Contract #

When you deploy an application into a Kubernetes cluster, you’re signing a bilateral “employment contract”:

The Application (Developer) promises Kubernetes:
  ✓ "My app needs at least 256Mi RAM and 0.25 CPU cores to boot."
  ✓ "My app is ready to serve user traffic when the HTTP `/readyz` endpoint succeeds."
  ✓ "My app is healthy when the HTTP `/healthz` endpoint responds successfully."
  ✓ "My app is uniquely identified by the `app=payment-api` label tag."
  ✓ "My app will handle the termination signal (SIGTERM) to close database connections."

Kubernetes (Platform) promises the Application:
  ✓ "I guarantee the minimum resource capacity you requested (requests) is always available."
  ✓ "I will stop sending traffic if your app reports itself as not ready."
  ✓ "I will automatically restart your container if it appears stuck (deadlock)."
  ✓ "I will route network traffic to the right Pods based on your identity selector."
  ✓ "I will give a tolerance window before force-killing your app during rolling updates."

If either party breaks its promises in this contract, system instability in production is unavoidable.

1. The Resource Allocation Contract: Requests, Limits, and QoS #

When you write resources.requests and resources.limits in a Pod manifest, you’re setting up a resource contract with the Scheduler and the Linux Kernel.

  • requests (Minimum Promise): The Kubernetes Scheduler uses this number to find a node. Kubernetes guarantees that RAM/CPU capacity equal to these requests is always locked exclusively for your container.
  • limits (Maximum Cap): The maximum compute limit the container is allowed to consume.

Consequences of Breaking the Resource Contract: #

If your container tries to use resources beyond the agreed limits, Kubernetes and the Linux kernel dish out different “punishments”:

  • Memory (RAM) Limit Violation: RAM is non-compressible. If physical RAM runs out, the system can’t be slowed down. So if a container consumes memory beyond limits.memory, the Linux kernel instantly kills the container process. The container dies with the OOMKilled (Out Of Memory Killed) status, and the Kubelet restarts it.
  • CPU Limit Violation: CPU is compressible. If a container consumes CPU beyond limits.cpu, the Linux kernel doesn’t kill the container. Instead, the kernel throttles the CPU capacity (CPU Throttling). The application stays alive, but its performance drops dramatically (causing API latency to balloon).

Cluster Quality of Service (QoS) Classes #

Kubernetes groups Pods into three QoS classes based on request/limit configuration to determine eviction priority when a node runs out of memory:

  1. Guaranteed (Highest Priority): Request and limit values for CPU and memory are explicitly written and numerically identical. These Pods are first-class citizens, most protected from eviction.
  2. Burstable: Request and limit values are configured but differ (limit is greater than request). The Pod gets evicted if the node runs out of memory after all BestEffort Pods are gone.
  3. BestEffort (Lowest Priority): We don’t write request and limit specs at all. This Pod gets no resource guarantees whatsoever and is the first target for force-kill when the node faces capacity pressure.

2. The Health & Availability Contract: Health Probes #

Kubernetes can’t guess whether your application is truly functional just by detecting that the container process is running. A Java or Node.js application can run error-free as a process while internally suffering memory leaks, thread deadlocks, or completely severed database connections.

For that reason, the application must provide dedicated endpoints for the Kubelet to check, via three types of Probes:

Liveness Probe (Death Detection) #

Answers the question: “Is this application still alive or completely stuck?”. If this probe fails several times in a row, the Kubelet kills the container and creates a new one (restart).

Readiness Probe (Traffic Readiness) #

Answers the question: “Is this application ready to accept user traffic?”. While the application is starting up, loading initial caches, or migrating, this probe must return failure. Kubernetes won’t restart the container if this probe fails; it only removes the Pod’s IP from the Service’s load balancer list so users don’t get 502/504 errors.

Startup Probe (Boot Protection) #

Made for heavy legacy applications that take a very long time to boot (e.g. 2-3 minutes). The startup probe blocks liveness and readiness probe monitoring until the container fully completes its first boot initialization.

ANTI-PATTERN: Identical Liveness & Readiness Endpoints
// WHAT WE DO:
- Configure the same endpoint (e.g. `/healthz`) for both liveness and readiness probes:
  livenessProbe: { httpGet: { path: /healthz } }
  readinessProbe: { httpGet: { path: /healthz } }
- Inside the application code, the `/healthz` endpoint is configured to check the connection to an external SQL Database.
// THE CONSEQUENCES IN PRODUCTION:
- When the database briefly overloads, the `/healthz` endpoint fails to respond.
- Kubernetes detects a failed liveness probe. Containers are immediately killed and massively restarted across the cluster (Cascading Restarts).
- This mass restart amplifies the network problem and extends service downtime, even though our application containers are actually perfectly healthy.

✓ THE RIGHT SOLUTION:
- Separate the endpoints:
  - **Liveness Probe** calls a lightweight endpoint (e.g. `/liveness`) that only verifies the internal web server is active and isn't tied to an external database connection.
  - **Readiness Probe** calls the `/readiness` endpoint, which actively tests connection readiness to the database, cache, and external API dependencies.
- If the database dies, traffic stops flowing to that Pod, but the Pod isn't restarted endlessly.

3. The Identity Contract: Labels & Selectors #

Kubernetes doesn’t use IP addresses to route network traffic to application Pods. Because Pod IPs are dynamic and can change at any time, Kubernetes uses Labels and Selectors as the identity contract for building relationships between objects.

# Service Manifest (Contract Selector)
apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-api # The Service tracks Pods with this label
  ports:
  - port: 80
    targetPort: 8080

As long as deployed Pods carry the app: payment-api label in their metadata, the Service object automatically adds that Pod’s IP address to its routing endpoint list. This loose coupling ensures we can release new application versions without manually updating load balancer configuration.

4. The Termination Contract: Graceful Shutdown #

The application lifecycle in Kubernetes is very dynamic. Pods are frequently deleted and recreated due to rolling updates, autoscaling down, or node maintenance drains.

When Kubernetes decides to shut down a Pod, it sends a standard Linux OS termination signal. The application must handle this signal properly to avoid cutting off user transactions mid-way.

Graceful Termination Workflow: #

  1. The Pod is moved to Terminating status: The Pod’s IP is removed from all Service endpoints so it stops receiving new traffic.
  2. The SIGTERM signal is sent: Kubernetes sends the SIGTERM signal (Signal 15) to the main process (PID 1) inside the container.
  3. Grace Period Wait: Kubernetes waits for the tolerance window defined in the manifest (default terminationGracePeriodSeconds: 30 seconds).
  4. SIGKILL Execution: If the application is still alive after the tolerance window expires, Kubernetes sends the SIGKILL signal (Signal 9) to force-kill the process.

Here’s an example of implementing SIGTERM signal handling at the application level (using Node.js as an example):

// CORRECT: Graceful Shutdown Handling of the SIGTERM Signal
const express = require('express');
const app = express();
const server = app.listen(8080);

process.on('SIGTERM', () => {
  console.log('SIGTERM signal received. Starting graceful server shutdown...');
  
  // 1. Stop accepting new requests
  server.close(() => {
    console.log('HTTP Server has been closed.');
    
    // 2. Close the database connection safely
    db.close().then(() => {
      console.log('Database connection closed successfully.');
      process.exit(0); // Exit with success code
    });
  });
});

An application that doesn’t handle the SIGTERM signal dies abruptly during rolling updates, leaving database transactions hanging and triggering connection errors for users.

Summary #

  • The Infrastructure Contract Is an Agreement — The application declares its needs and behaviors, and Kubernetes uses that information to manage the cluster autonomously.
  • CPU Throttling vs OOMKilled — Violating the memory limit (RAM limit) ends with the container force-killed (OOMKilled), while violating CPU only slows the container down (throttling).
  • Correct Probe Semantics — The liveness probe triggers restarts for stuck containers, while the readiness probe controls user traffic routing (don’t use a database endpoint for liveness).
  • Graceful SIGTERM Shutdown — Make sure your application code catches the SIGTERM signal to finish in-flight transactions and close database connections before dying.
  • terminationGracePeriodSeconds — Always tune the shutdown timeout in the cluster manifest if your application needs a long time to clean up transient in-memory data.

← Previous: Configuration   Next: Overview →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact