API Server #

The API Server (kube-apiserver) is the main gateway of a Kubernetes cluster. Acting as both the cluster’s receptionist and security guard, the API Server is the only component that exposes the cluster control API to the outside world. No cluster state change can happen without the API Server’s approval and processing.

For DevOps teams and developers, understanding the API Server’s internal processing flow is crucial for tracking permission issues (RBAC), designing security policy webhook integrations, and optimizing application-to-cluster interaction performance.

API Server Request Processing Pipeline #

Every time we send a command (e.g. kubectl apply -f deployment.yaml, or when the Kubelet reports its health status), the request must pass through a strict series of checks inside the API Server before finally being stored in the etcd database.

Here’s a visualization of the request processing pipeline inside the API Server:

flowchart TD
    Req["Incoming Request (HTTP POST/PUT/DELETE)"] --> AuthN{"1. Authentication\n(Who Is Sending It?)"}
    
    AuthN -- Failed (401 Unauthorized) --> Terminate["Stop Request & Return Error"]
    AuthN -- Success --> AuthZ{"2. Authorization\n(Are They Allowed?)"}
    
    AuthZ -- Failed (403 Forbidden) --> Terminate
    AuthZ -- Success --> Mutating{"3. Mutating Admission\n(Modify/Complete the Object)"}
    
    Mutating --> SchemaVal{"4. Schema Validation\n(Is the YAML Structure Valid?)"}
    
    SchemaVal -- Failed (422 Unprocessable) --> Terminate
    SchemaVal -- Success --> Validating{"5. Validating Admission\n(Is It Allowed to Be Created?)"}
    
    Validating -- Failed (403 Forbidden) --> Terminate
    Validating -- Success --> EtcdWrite["6. etcd Write\n(Store Permanently in DB)"]
    
    EtcdWrite --> Response["Send HTTP 201 Created to Client"]

    style AuthN stroke:#2980b9,stroke-width:2px
    style AuthZ stroke:#f39c12,stroke-width:2px
    style Mutating stroke:#8e44ad,stroke-width:2px
    style SchemaVal stroke:#7f8c8d,stroke-width:2px
    style Validating stroke:#d35400,stroke-width:2px
    style EtcdWrite stroke:#27ae60,stroke-width:2px

Pipeline Stage Explanations: #

1. Authentication (Who Is Sending It?) #

The API Server verifies the sender’s identity using TLS client certificates, HTTP Bearer tokens, or external authentication (like OpenID Connect). If the identity is invalid, the request is immediately rejected with HTTP status code 401 Unauthorized.

2. Authorization (Are They Allowed?) #

Once the identity is recognized, the API Server checks whether that user has permission to perform the requested action on the target object (e.g. “is user John allowed to delete Pods in the production namespace?”). This evaluation is resolved through RBAC (Role-Based Access Control) policies. Failure at this stage produces HTTP status 403 Forbidden.

3. Mutating Admission Webhooks (Object Modification) #

The webhook stage where the API Server sends the manifest to an external system for completion. Mutating webhooks can dynamically change manifest contents — for example, automatically injecting a Fluentd sidecar container into a Pod, or adding default environment variables to containers.

4. Schema Validation #

The API Server verifies whether the submitted YAML/JSON manifest data structure complies with the official Kubernetes spec (e.g. checking for field name typos in the YAML).

5. Validating Admission Webhooks (Policy Validation) #

Unlike mutating webhooks that complete objects, Validating Webhooks reject requests that violate cluster organization policies (for example, rejecting Pod creation if the container image doesn’t come from the company’s internal registry, or rejecting containers trying to run as root).

6. etcd Write (Permanent Storage) #

After passing all the check stages above, the API Server finally decrypts the data (if it’s a Secret object) and writes the object permanently to the etcd database. Only at this stage is the cluster state officially considered changed.

The API Server Watch Mechanism: gRPC/HTTP2 Streaming Efficiency #

Kubernetes heavily depends on information propagation speed. When a new state appears in etcd (e.g. a Pod must run on Node-2), the Scheduler and the Kubelet on Node-2 must learn about it within milliseconds.

If components used polling (asking continuously), the cluster database would collapse under the load flood. That’s why the API Server uses the Watch API.

The Watch mechanism works by leveraging the HTTP/2 protocol (or WebSockets), which supports long-lived two-way streaming connections (persistent connections).

  • The Kubelet opens one Watch connection to the API Server and declares: “Tell me if there are any new Pod changes assigned to my node”.
  • The API Server keeps that connection open.
  • As soon as the Scheduler writes the Pod’s node assignment to etcd, the API Server detects the change and immediately pushes the update event as a stream of small data packets through the open persistent connection to the Kubelet in real-time.

Admission Controllers in Production: Enforcing Policy #

Admission Controllers are plugins running inside the API Server process to restrict and complete cluster resources. In modern production clusters, we often use external policy engines like OPA Gatekeeper or Kyverno, integrated through Validating Admission Webhooks.

Example production use cases of Admission Controllers:

  • Preventing deployments of applications that don’t define resources.requests and limits, to avoid exhausting node capacity.
  • Ensuring every Pod carries the mandatory team and environment labels for audit purposes.
  • Automatically adding TLS certificates to Ingresses using the cert-manager mutating webhook.
ANTI-PATTERN: Bypassing the API Server to Modify etcd Directly
// WHAT WE DO:
- Connect the `etcdctl` utility directly to the cluster's etcd database to forcibly edit
  or delete Kubernetes object entries to resolve a stuck Pod problem.
// THE CONSEQUENCES IN PRODUCTION:
- Broken Schema Validation: We bypass the entire API Server validation pipeline. If we write an entry
  incorrectly, etcd stores corrupt data that crashes the kube-apiserver process during boot.
- Cluster Inconsistency: The Controller Manager doesn't get watch event notifications,
  disorienting the reconciliation loop about cluster state.
- Total Data Loss Risk: Manual etcd manipulation without API Server control can break Raft quorum.

✓ THE RIGHT SOLUTION:
- Always interact with the cluster through the API Server using `kubectl` commands or the official API.
- If an object is stuck and can't be deleted (e.g. because of finalizers), use the official API patch to clean it:
  `kubectl patch pod <pod-name> -p '{"metadata":{"finalizers":null}}'`
- Let the API Server manage all etcd database writes safely and in a controlled manner.

Summary #

  • The Cluster’s Central Receptionist — The API Server is the main gateway; all internal components and external tools must communicate through it.
  • Strict Validation Pipeline — Every request must pass Authentication ➔ Authorization (RBAC) ➔ Mutating Webhook ➔ Schema Validation ➔ Validating Webhook before being written to etcd.
  • Watch via HTTP/2 — The API Server propagates state change events in real-time using HTTP/2 gRPC streaming connections, eliminating polling overhead.
  • Policy Enforcement — Admission Controllers (like Kyverno/Gatekeeper) secure production clusters by rejecting manifests that violate organizational rules.
  • No etcd Bypass — Never try to modify the etcd database directly without going through the API Server to avoid corrupting the cluster database.

← Previous: Worker Node   Next: Scheduler →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact