Managed Kubernetes #

Building and operating a Kubernetes cluster self-managed on top of bare-metal infrastructure or raw VMs is a very heavy operational task. We’re fully responsible for the entire infrastructure lifecycle: designing high-performance distributed etcd clusters, monitoring Control Plane component health statuses (API Server, Controller Manager, Scheduler), managing periodic internal CA security certificate rotation, and doing OS version upgrades on nodes without triggering service downtime. Managed Kubernetes comes to remove most of this administrative operational burden. By delegating Control Plane management to cloud providers, we can shift 100% of our focus to developing and managing our business applications (workloads). This article deeply discusses the shared responsibility model, a comparison of the three giant platforms (GKE, EKS, AKS), vendor lock-in prevention strategies, and best practices for managing production clusters in the cloud.

The Shared Responsibility Model #

It’s important to understand that switching to Managed Kubernetes doesn’t mean we’re free of all infrastructure responsibilities. Cloud providers apply a Shared Responsibility Model that strictly divides tasks between the provider and the consumer.

+-----------------------------------------------------------------------+
|                       CONSUMER RESPONSIBILITIES                       |
|   - Workload Configuration (Deployment, Pod, Ingress, Network Manifests)|
|   - RBAC (Role-Based Access Control) & NetworkPolicy Rules            |
|   - Container Image & Application Security                            |
|   - Node Pool Capacity Management & Autoscaling Schemes               |
|   - Kubelet Version Updates on Worker Nodes (except Serverless)       |
+-----------------------------------------------------------------------+
|                        PROVIDER RESPONSIBILITIES                      |
|   - API Server High Availability                                      |
|   - Consistent etcd Storage, Periodic Backups, & Data Encryption      |
|   - Automatic Security Certificate Rotation on the Control Plane      |
|   - Health Monitoring & Automatic Master Component Replacement        |
+-----------------------------------------------------------------------+

With this model, if the API Server suffers access failures, that’s the cloud provider’s fault (covered under their SLA guarantee). However, if our application gets hacked because we left database ports open to the public internet through a wrong LoadBalancer Service configuration, that’s entirely our responsibility.

Google Kubernetes Engine (GKE) #

As the pioneer that released Kubernetes to the open-source world, Google Cloud has the GKE service often considered the gold standard of Managed Kubernetes platforms with the most mature Kubernetes-native feature integration.

1. Autopilot vs Standard Modes #

GKE offers two main operating modes determining our infrastructure control level:

• GKE Autopilot (Zero Node Management): Google manages the entire cluster, including worker node provisioning, scaling, and security hardening automatically. We don’t need to create node pools or choose VM types (like n2-standard-4). We just send application manifests, and Google dynamically matches node CPU/RAM capacity according to Pod needs. We’re only billed based on the total resource requests of running Pods, not physical Node capacity.
• GKE Standard (Full Control): We have full access to node pool configurations. We can SSH into Nodes, install third-party DaemonSets accessing host kernels, and design custom VM architectures. However, we’re responsible for patching node OSes and managing node pool lifecycles.

2. Workload Identity: Keyless Authentication #

Storing Google Service Account access credential files in JSON form inside Kubernetes Secrets is very dangerous because they’re prone to leaking. GKE solves this through Workload Identity.

flowchart LR
    PodSA["1. Pod with a Kubernetes ServiceAccount (KSA)"] -->|"Send the KSA Token"| GKE_OIDC["2. GKE OIDC Provider (Reads Identity)"]
    GKE_OIDC -->|"Validate & Exchange the Token"| GCP_STS["3. GCP Security Token Service (STS)"]
    GCP_STS -->|"Return a Temporary GCP Token"| GCP_IAM["4. GCP IAM (Google Service Account - GSA)"]
    GCP_IAM -->|"Access the Allowed Resources"| GCPResources["5. GCP Resources (Cloud Storage, BigQuery)"]

With Workload Identity, KSAs inside the Kubernetes cluster are directly mapped (federated) to GSAs at the GCP IAM level using short-lived OIDC tokens automatically rotated hourly by Google STS, so no permanent credential key files are stored inside the cluster.

# Example of registering a Workload Identity mapping via the gcloud CLI
gcloud iam service-accounts add-iam-policy-binding [email protected] \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project.svc.id.goog[production/billing-ksa]"

Then, on the Kubernetes side, we just create a ServiceAccount with an annotation referencing that Google Service Account:

# File: gke/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: billing-ksa
  namespace: production
  annotations:
    iam.gke.io/gcp-service-account: "[email protected]"

Amazon Elastic Kubernetes Service (EKS) #

Amazon EKS is AWS’s Managed Kubernetes service. Given AWS’s market dominance, EKS is the most widely used platform at large enterprise levels, especially for those needing deep integration with other AWS service ecosystems.

1. IRSA (IAM Roles for Service Accounts) #

IRSA is the AWS identity federation implementation equivalent to Workload Identity on GKE. IRSA lets application containers inside Kubernetes pods authenticate directly to AWS APIs (like reading data from S3 buckets or writing to DynamoDB) without storing AWS_ACCESS_KEY_ID credential files inside Secrets.

When we enable IRSA, EKS acts as an OpenID Connect (OIDC) identity provider. The AWS Security Token Service (STS) verifies the Kubernetes service account token, then dynamically provides short-lived AWS IAM credentials into the container.

# File: eks/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader-ksa
  namespace: production
  annotations:
    # Binds the Kubernetes ServiceAccount directly to an AWS IAM Role
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/prod-s3-reader-role"

2. AWS Fargate: Serverless Kubernetes #

For organizations wanting a serverless approach without managing EC2 (Elastic Compute Cloud) instances as worker nodes, EKS provides integration with AWS Fargate.

Every time we deploy a Pod into a namespace configured with a Fargate profile, AWS launches a dedicated isolated micro virtual machine (microVM) to run that Pod. We don’t need to do OS patching on nodes, and we avoid the security threat of container-to-host-kernel privilege escalation because every Pod has its own physical VM boundary.

Azure Kubernetes Service (AKS) #

Microsoft Azure provides AKS with a main focus on cluster provisioning speed and strong enterprise integration using Microsoft Entra ID (formerly Azure Active Directory).

1. Native Microsoft Entra ID Integration #

One of AKS’s biggest advantages is its ability to connect Kubernetes RBAC (Role-Based Access Control) authorization systems directly with company Microsoft Entra ID accounts.

# Example of AKS authorization configuration integrated with Azure AD
# We don't need to create manual RoleBindings for every user; 
# we just map an Azure AD Group ID directly to the Kubernetes admin ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: active-directory-admin-binding
subjects:
- kind: Group
  name: "88888888-4444-4444-4444-121212121212" # The Azure AD Object Group ID for the Platform Team
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

With this configuration, when developers want to interact with the cluster using kubectl, they must do browser login authentication using official company emails and comply with the Multi-Factor Authentication (MFA) security rules applied at the company AD level.

2. Virtual Nodes (Azure Container Instances - ACI) #

AKS provides a unique feature called Virtual Nodes leveraging Azure Container Instances (ACI). This feature is very useful for handling unexpected traffic surges (burst traffic).

When detecting high Pod queues due to node pool CPU capacity limits, AKS Virtual Nodes can launch new containers in ACI within seconds to absorb that instant traffic load, while the main cluster waits for the physical VM node auto-scaling process which usually takes 2 to 4 minutes to start.

Platform Comparison Table #

The following table presents an in-depth comparative analysis between the three largest Managed Kubernetes platforms in the market:

Vendor Lock-in Prevention Strategies #

Although Managed Kubernetes eases cluster operations, uncontrolled use of exclusive cloud provider features can lock our system into one specific vendor. If drastic infrastructure rental cost increases happen later or there’s a migration need to another cloud provider, we’ll be trapped in an expensive, time-consuming migration process.

We can apply the following strategies to minimize specific vendor dependencies:

flowchart TD
    AppManifests["Main Application (Deployments, Services, ConfigMaps)"] -->|"Agnostic"| PortableLayer["Agnostic Kubernetes API (Portable to All Clouds)"]
    
    subgraph CloudSpecificOverlays["Kustomize Overlays per Cloud Vendor"]
        GCPOverlay["GCP Overlay (GKE StorageClass, Google Ingress Controller)"]
        AWSOverlay["AWS Overlay (EKS StorageClass, EBS annotations)"]
        AzureOverlay["Azure Overlay (AKS StorageClass, Azure LoadBalancer annotations)"]
    end
    
    AppManifests -.-> GCPOverlay
    AppManifests -.-> AWSOverlay
    AppManifests -.-> AzureOverlay

1. Use Kustomize Overlays for Cloud-Specific Metadata #

Separate our main application manifests from cloud-specific annotation configurations. Put agnostic specs in the base/ directory, then use Kustomize overlays/ to insert load balancer annotations or storage class specs varying between cloud providers.

# File: k8s/overlays/aws/service-patch.yaml
# A special patch to enable the AWS Network Load Balancer (NLB)
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

2. Standardize Using Agnostic Operators #

Instead of using cloud provider built-in custom operators or services (like the AWS Controller for Kubernetes or GCP Config Connector) to create SQL databases, use agnostic operators like Crossplane. Crossplane abstracts cloud provider resources into neutral Kubernetes objects easily movable between clusters.

3. Use cert-manager and external-dns #

Don’t rely on cloud provider built-in automatic DNS mechanisms fused with their Ingress control. Use open-source tools like cert-manager (for TLS certificates) and external-dns (for automatic DNS name synchronization to Route53, Cloud DNS, or Azure DNS from neutral Ingress objects).

Best Practices for Production Managed Kubernetes #

Apply the following configuration guides to make sure your cloud production cluster runs safely, efficiently, and with high fault tolerance:

1. Enable Private Clusters (Network Isolation) #

Never let your production cluster’s Kubernetes API Server endpoint be openly accessible from the public internet. Enable the Private Cluster feature.

• The API Server only has an internal IP inside the VPC.
• Administrative access from engineer laptops must go through office VPN paths, AWS Client VPN, Bastion Hosts, or use Cloud NAT / Private Service Connect interconnections.
• Worker nodes are placed in private subnets without external public IP addresses.

2. Design Multi-Availability Zone (Multi-AZ) Node Pools #

To protect the system from physical data center failures in one region, spread your cluster worker nodes across at least three different Availability Zones. Use the topologySpreadConstraints property on Pod manifests to ensure Kubernetes evenly divides container replicas across those zones.

# File: app/deployment.yaml (Topology spread code snippet)
spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: payment-api

3. Separate Node Pools by Workload Role #

Don’t combine all cluster applications and services into one big common node pool. Create several dedicated node pools based on workload characteristics:

• System Node Pool: Specifically for running internal cluster components (CoreDNS, metrics-server, Ingress Controller). This pool must have high fault tolerance and use fully paid VMs (on-demand VMs).
• Application Node Pool: For running our business microservice application code.
• Spot / Preemptible Node Pool: A node pool using cheap spare VMs (saving up to 70-80% of cloud costs). This pool is great for running testing applications (dev/staging) or batch processing jobs insensitive to sudden termination processes.

Managed Kubernetes Practice Anti-Patterns #

Avoid the following operational mistakes to prevent cloud cost waste and cluster security holes:

1. Storing Permanent Credential Keys Inside the Cluster #

# ANTI-PATTERN: Storing a permanent Google Service Account JSON key inside a Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: gcp-sa-key
type: Opaque
data:
  key.json: "ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgIC..." # DON'T: Key leak risk!

Security Leak Risks:
- If someone hacks one Pod and gets Secret read access, they can take that JSON key and use it outside the cluster to delete valuable data in company cloud storage.
✓ SOLUTION: Always use cloud provider native identity federation (GKE Workload Identity / EKS IRSA / Azure Workload Identity) which requires no permanent key file storage in the cluster at all.

2. Running Clusters Without Autoscaler Limits (Cost Ballooning) #

Enabling the Cluster Autoscaler feature while leaving the maximum node capacity parameter unlimited or set too high without cost monitoring.

# ANTI-PATTERN: Setting max-nodes too large without calculating company budget limits
eksctl create cluster --name prod --nodes-min 3 --nodes-max 100 # DON'T: Potential bloated cloud bills!

If our application suffers a denial-of-service (DDoS) attack or has a memory leak bug (memory leak loop) triggering the HPA to keep adding Pods, the Cluster Autoscaler keeps launching new VM instances until the 100-node maximum limit. This triggers cloud bill ballooning (cost ballooning) up to thousands of dollars in one night.

✓ SOLUTION:
Limit the maximum node pool capacity rationally based on monthly budgets and target physical capacity (e.g. setting max-nodes to 15). Also enable budget alerting systems (Cloud Billing Alerts) for instant notifications if infrastructure cost consumption spikes.

Managed Kubernetes Audit Checklist #

Use the following checklist before releasing your Managed Kubernetes cluster to the production stage:

NETWORK & CONNECTIVITY HARDENING:
  □ The cluster is configured as a Private Cluster (the API Server isn't exposed to the public internet).
  □ The API Server endpoint is protected using IP restriction rules (Authorized Networks / AWS Security Groups).
  □ Worker nodes are placed in private subnets without external public IPs.
  □ Pod-to-pod traffic is restricted using NetworkPolicy rules.

IDENTITY & IAM SECURITY INTEGRATION:
  □ No permanent credential key files (like JSON keys or AWS Access Keys) are stored in the cluster.
  □ Identity federation features (Workload Identity / IRSA) are active and used by all pods needing cloud resource access.
  □ IAM Role access rights are restricted using the least privilege principle for each ServiceAccount.
  □ Secret data encryption in etcd uses cloud provider built-in KMS (AWS KMS / GCP KMS / Azure Key Vault).

FAULT TOLERANCE & COSTS:
  □ Worker nodes are spread across at least three different Availability Zones (Multi-AZ).
  □ Node pools are separated between System Pools (On-Demand) and Application/Batch Pools (Spot VM options).
  □ Minimum and maximum Cluster Autoscaler parameters are rationally configured to limit spending.
  □ Maintenance policies (Maintenance Windows) are configured during quiet user traffic hours.

Summary #

• Delegate the Master Plane — Use Managed Kubernetes to move Control Plane administrative responsibilities (etcd, API Server, CA rotation) to cloud providers for team efficiency.
• Integrate Native Identity — Leave static key storage behind in clusters; must use Workload Identity (GKE) or IRSA (EKS) for secure keyless authentication.
• Implement Private Clusters — Close API Server access from the public internet to minimize the cyber attack surface on production clusters.
• Spread Nodes Across AZs — Make sure your cluster has regional-level disaster tolerance by spreading node pools across at least three separate Availability Zones.
• Minimize Lock-in from the Start — Abstract cloud provider specific metadata using Kustomize overlays so your main manifests stay agnostic and easily movable.
• Control Autoscaling — Configure the Cluster Autoscaler maximum capacity upper limit rationally to prevent unexpected cloud bill spikes from DDoS attacks or application bugs.

← Previous: Operator Pattern   Next: Ecosystem & Tooling Anti-Patterns →