Kustomize #

Managing Kubernetes manifests for various deployment environments (like development, staging, and production) often ends in file duplication problems. We’re often tempted to copy entire YAML manifest files from one environment directory to another, then manually change a few configuration lines. This copy-paste practice is very risky because when updates happen to the base configuration (e.g. adding a sidecar container or changing health check probes), we must update all those duplicated files one by one. Kustomize comes to solve this problem with a template-free approach. As a built-in configuration management tool integrated directly into kubectl since version 1.14, Kustomize lets us define the base configuration once (base), then create environment-specific customizations (overlays) through pure declarative decoration and patching mechanisms without damaging the original YAML files.

The Template-Free Philosophy #

Most package management tools like Helm use text template engines (like Go Templates) inserting dynamic variables into manifests using special markers (like {{ .Values.replica }}). This approach has several limitations:

• Syntactically Invalid YAML: Helm template files before compilation aren’t valid YAML. This means we can’t validate them using standard YAML parsers or IDE linter tools before the rendering process finishes.
• Logic Complexity: Over time, templates tend to fill up with complicated conditional logic (if/else) and loops (range). This makes manifests hard to read and maintain for team members unfamiliar with those template syntaxes.
• Injection Risk: Spacing or data type handling errors during text rendering can cause manifest parsing failures or configuration injection security holes.

Kustomize takes a very different approach. All configuration files in Kustomize are valid, pure Kubernetes YAML files. Kustomize doesn’t do text interpolation; instead, it reads those YAML documents into memory as an object tree data structure, does structured transformations (like renaming, adding labels, or overwriting property values), then prints back clean YAML manifests.

Kustomize Architecture and Workflow #

Kustomize works by processing the kustomization.yaml file defining the base resources, generators, transformers, and patch rules to be applied. The Kustomize compilation workflow is described in the following diagram:

flowchart TD
    BaseDir["Base Directory (base/)"] -->|"resources"| KustEngine["Kustomize Engine"]
    OverlayDir["Overlay Directory (overlays/production/)"] -->|"resources & bases"| KustEngine
    
    subgraph EnginePipeline["Kustomize Engine Pipeline"]
        direction TB
        GeneratorStage["1. ConfigMap/Secret Generators"] --> TransformerStage["2. Built-in Transformers (labels, annotations, namespace)"]
        TransformerStage --> PatchStage["3. Patching Phase (Strategic Merge & JSON Patches)"]
    end
    
    KustEngine --> EnginePipeline
    EnginePipeline -->|"Output Manifests"| FinalYAML["Final Plain YAML Manifests"]
    FinalYAML -. "kubectl apply -k" .-> K8sAPI["Kubernetes API Server"]

When we run the Kustomize command, the internal pipeline works in the following order:

1. Generator Stage: Kustomize creates new resources like ConfigMaps and Secrets based on specified text files, properties, or literals. The generated objects are inserted into the manifest list with unique hash suffixes added to their names.
2. Transformer Stage: Kustomize applies global transformation rules hierarchically. This includes changing the target namespace, adding resource name prefixes/suffixes, adding global labels (commonLabels), and rewriting container image tags (image tag overrides).
3. Patching Stage: Kustomize overwrites or inserts specific properties into certain objects using the Strategic Merge Patch or JSON Patch (RFC 6902) methods. The final output of this process is pure YAML manifests ready to be sent to the API Server.

Standard Production Directory Structure #

To implement Kustomize cleanly, we must strictly separate the base code (base) containing the general application architecture from variant code (overlays) containing configurations specific to each operational environment.

k8s/
├── base/                         # The base configuration inherited by all environments
│   ├── kustomization.yaml        # The list of managed base resources
│   ├── deployment.yaml           # The vanilla deployment manifest (no environment modifications)
│   ├── service.yaml              # The application's internal network port abstraction
│   └── configmap.yaml            # The default application configuration framework
└── overlays/                     # Environment-specific customizations
    ├── development/              # The Development environment variant
    │   ├── kustomization.yaml    # Connect to the base & set small-scale overrides
    │   └── deployment-patch.yaml # Patch resource limits and dev env
    ├── staging/                  # The Staging environment variant
    │   ├── kustomization.yaml
    │   └── deployment-patch.yaml
    └── production/               # The Production environment variant (Large Scale)
        ├── kustomization.yaml
        ├── deployment-patch.yaml # High resource limit configuration & many replicas
        └── hpa.yaml              # HPA resources (only active in the production environment)

Implementing Base Components #

Let’s define the base components (base) first. Files inside the base/ folder must not be directly modified when we only want to change configuration values for a specific environment.

1. base/kustomization.yaml #

This file acts as an index file telling Kustomize which manifest files are part of our base application.

# File: k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Declares all Kubernetes manifests managed at the base level
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

# Adds standard labels to all base resources for easy tracking
commonLabels:
  app.kubernetes.io/part-of: core-banking
  app.kubernetes.io/component: transaction-api

2. base/deployment.yaml #

This is our application Deployment blueprint. We write the standard container spec here.

# File: k8s/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transaction-api # The vanilla name without environment prefixes/suffixes
spec:
  replicas: 1           # The default minimum replica count
  selector:
    matchLabels:
      app: transaction-api
  template:
    metadata:
      labels:
        app: transaction-api
    spec:
      containers:
      - name: app-container
        image: registry.company.com/banking/transaction-api:latest
        ports:
        - containerPort: 8080
          name: http
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 300m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10

3. base/service.yaml #

The base network port service abstraction for connecting internal cluster traffic.

# File: k8s/base/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: transaction-api
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: transaction-api

4. base/configmap.yaml #

The non-sensitive base application configuration framework.

# File: k8s/base/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_HOST: "localhost"
  DB_PORT: "5432"
  LOG_LEVEL: "info"

Implementing Overlay Components (The Production Environment Case) #

Now we’ll write the configuration variant for the production environment. In this environment, we want to increase the replica count to 5, configure higher container resource limits, point the application to the external production database, and add the Horizontal Pod Autoscaler (HPA) feature not needed in the development environment.

1. overlays/production/kustomization.yaml #

This overlay-level customization file imports the base components and applies specific modification rules.

# File: k8s/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Points the relative path to the base directory
resources:
  - ../../base
  - hpa.yaml # Add the HPA resource that only exists in production

# Changes the target namespace automatically for all resources
namespace: prod-banking

# Gives resource names a prefix to visually distinguish them in the cluster
namePrefix: prod-
# Result: transaction-api -> prod-transaction-api

# Overrides the container image tag to the stable production release version
images:
  - name: registry.company.com/banking/transaction-api
    newTag: v1.4.2

# Adds environment-specific labels globally
commonLabels:
  environment: production
  tier: backend

# Adds change tracking annotations to all manifests
commonAnnotations:
  kubernetes.io/change-cause: "Release update to version v1.4.2 for high performance"

# Applies specific modification patches to the Deployment object
patches:
  - path: deployment-patch.yaml
    target:
      kind: Deployment
      name: transaction-api

2. overlays/production/deployment-patch.yaml #

Inside this patch file, we only write the YAML structure parts we want to change or add. We don’t need to rewrite the entire base Deployment spec.

# File: k8s/overlays/production/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transaction-api # Must match the resource name in the base
spec:
  replicas: 5           # Raise replicas to 5 for high availability in production
  template:
    spec:
      containers:
      - name: app-container # Kustomize matches containers by name
        resources:
          # Raise resource limits to handle production traffic
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 2000m
            memory: 2Gi
        env:
          - name: APP_ENV
            value: "production"
          - name: DB_HOST
            value: "prod-db-cluster.internal.company.com" # Point to the production DB

3. overlays/production/hpa.yaml #

An additional resource only enabled in the production environment for dynamic auto-scaling.

# File: k8s/overlays/production/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: transaction-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: prod-transaction-api # Note the name must account for the prod- prefix
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75

Patching Mechanisms in Depth #

Kustomize provides two main ways to customize (patch) base manifests: Strategic Merge Patch and JSON Patch (RFC 6902). Understanding the difference between them is crucial when designing complex configuration changes.

1. Strategic Merge Patch (SMP) #

Strategic Merge Patch is Kubernetes’s built-in way of merging two YAML documents. This mechanism works intelligently by analyzing the Kubernetes object schema. SMP uses a special attribute called the merge key to match elements inside lists (arrays/lists), instead of overwriting the entire list like standard JSON processing.

For example, on a Deployment object, the container list (containers) uses name as the merge key.

# BASE SPECIFICATION:
spec:
  template:
    spec:
      containers:
      - name: app-container
        image: app:latest
      - name: log-exporter
        image: fluentd:v1

# PATCH SPECIFICATION:
spec:
  template:
    spec:
      containers:
      - name: app-container # SMP finds the container with the same name
        image: app:v1.4.2   # Only this line is changed

If Kustomize detects a merge key match, it merges that container’s internal properties (like replacing the image or adding environment variables). The second container element (log-exporter) stays unchanged even though it isn’t rewritten in the patch file.

2. JSON Patching (RFC 6902) #

JSON Patch is an industry standard for manipulating structured documents using very precise operation instruction lists like add, remove, replace, move, and copy. This approach is very useful when we want to change properties without merge keys (e.g. changing the first port in an unnamed port array, or removing certain volume elements).

To apply JSON Patches, we write the operation list directly in kustomization.yaml under the patches block:

# Example JSON Patch in kustomization.yaml
patches:
  - target:
      kind: Deployment
      name: transaction-api
    patch: |-
      # Operation 1: Replacing replicas directly
      - op: replace
        path: /spec/replicas
        value: 3
      
      # Operation 2: Removing the readiness probe from the first container (index 0)
      - op: remove
        path: /spec/template/spec/containers/0/readinessProbe
      
      # Operation 3: Inserting a new environment variable at the end of the env array
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: NEW_PROD_API_KEY
          value: "secure-prod-token-value"      

Configuration Generators: ConfigMap and Secret #

One of the biggest challenges in operating applications in Kubernetes is updating configuration without stopping services. By default, if we update data inside an external ConfigMap and apply those changes with kubectl apply, Kubernetes doesn’t trigger automatic restarts on Pods referencing that ConfigMap if it’s mounted as environment variables. As a result, our application keeps running with the old configuration until the Pod is manually deleted or restarted for other reasons.

Kustomize elegantly solves this problem through ConfigMapGenerator and SecretGenerator.

# File: k8s/overlays/production/kustomization.yaml (Generator code snippet)
configMapGenerator:
  - name: prod-app-config
    behavior: replace # Replaces the base configmap named 'app-config'
    files:
      - configs/app.properties # Reads configuration from an external file
    literals:
      - DB_HOST="prod-db-cluster.internal.company.com"
      - LOG_LEVEL="error"

secretGenerator:
  - name: prod-db-secrets
    type: Opaque
    literals:
      - DB_PASSWORD="SuperSecretProductionPasswordInput!"

Hash Suffix Mechanism and Automatic Rolling Updates #

When we run the Kustomize compilation process, these generators read the file or literal contents, then create new ConfigMap objects with a unique content hash inserted at the end of their names (e.g. prod-app-config-7hgk9m2x).

Contents of the configs/app.properties file:
  max_connections=100  --> Compilation --> ConfigMap: prod-app-config-7hgk9m2x

Change the configs/app.properties file values:
  max_connections=250  --> Compilation --> ConfigMap: prod-app-config-g8k2m9x4

Dynamically, Kustomize scans all our Deployment manifests referencing prod-app-config and rewrites those reference names to match the name containing the new hash (prod-app-config-g8k2m9x4). Because the ConfigMap name spec inside the Deployment object changes, the Kubernetes API Server detects this as a valid pod template update and automatically triggers the Rolling Update process to replace old Pods with new Pods carrying the latest configuration.

Main Built-in Transformers #

Kustomize provides built-in transformation tools (transformers) to speed up mass metadata modifications without writing repetitive patch files.

1. namespace #

This transformer moves all resources declared in the customization file to a specific namespace. If that namespace doesn’t exist in the base manifests, Kustomize automatically inserts it into all objects supporting namespace scope (namespaced-scope resources) like Pods, Services, Deployments, and Ingress, and updates RBAC binding references (like RoleBindings).

namespace: prod-finance-services

2. commonLabels & commonAnnotations #

Adds labels or annotations to all resources and matching selectors automatically. This greatly helps standardize cluster governance. Kustomize ensures new labels are also inserted into spec/selector/matchLabels and spec/template/metadata/labels on Deployments so Pod relationships aren’t broken.

commonLabels:
  billing-code: "dept-908"
  compliance: "pci-dss"

3. namePrefix & nameSuffix #

Inserts text prefixes or suffixes into the metadata/name column of all objects. This is very useful if we deploy the same application multiple times inside the same namespace for parallel testing purposes.

namePrefix: test-
nameSuffix: -v2
# The 'api' Service object is renamed to 'test-api-v2'

4. images #

Changes registry repository names, release tags, or container image SHA digest values without needing to manually search container paths inside Deployment manifests.

images:
  - name: registry.company.com/banking/transaction-api
    newName: registry-backup.company.com/banking/transaction-api-mirror
    newTag: v1.4.2-patch1

Kustomize Operational Commands #

We can run the Kustomize compilation process directly using the built-in kubectl CLI or using the standalone binary kustomize.

1. Preview Rendered Manifests #

Before applying manifests to the production cluster, run the render command to print the YAML compilation results to the console to make sure there are no spacing structure or patching errors.

# Preview the results using built-in kubectl
kubectl kustomize k8s/overlays/production/

# Preview the results using the standalone kustomize binary (if installed)
kustomize build k8s/overlays/production/

2. Checking Configuration Differences (Diff) #

To validate change impacts before modifying the cluster state, use the diff flag to see which lines will change.

# Compare the active cluster state with the new local manifests
kubectl diff -k k8s/overlays/production/

3. Applying Configurations Directly #

If the compilation and diff results are safely validated, send the manifest payload to the API Server using the customization directory option (-k).

# Apply the production environment configuration
kubectl apply -k k8s/overlays/production/

# Remove all resources managed by that customization
kubectl delete -k k8s/overlays/production/

Comparative Analysis: Kustomize vs Helm #

Both tools have different strengths and usage scenarios. We must choose the tool matching our application architecture characteristics and team operational capabilities.

Collaboration Scenarios (Helm + Kustomize Post-Rendering) #

At large enterprise production levels, we don’t have to choose only one of these two tools. We can combine the strengths of both using the Post-Rendering technique.

In this scenario, we use Helm to download and do the initial rendering of complex third-party packages available on the internet (e.g. the Bitnami PostgreSQL database), then pipe the rendered YAML output to Kustomize to add internal company compliance labels or do special security patches before sending it to the Kubernetes API Server.

# Render the public helm chart -> pipeline to local kustomize -> apply to the cluster
helm template my-postgres oci://registry-1.docker.io/bitnamicharts/postgresql \
  --values values-custom.yaml \
  | kubectl kustomize ./kustomize-patches \
  | kubectl apply -f -

Kustomize Implementation Anti-Patterns #

Avoid the following configuration design mistakes so our teams don’t get trapped in new hard-to-maintain complexity problems:

1. Copying the Entire Manifest Contents from Base to Overlays (Copy-Paste Anti-Pattern) #

# k8s/overlays/production/deployment-patch.yaml
# ANTI-PATTERN: Rewriting the entire base/deployment.yaml file contents
# just to change a replica line or resource limits.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transaction-api
spec:
  replicas: 5
  selector:
    matchLabels:
      app: transaction-api
  template:
    metadata:
      labels:
        app: transaction-api
    spec:
      containers:
      - name: app-container
        image: registry.company.com/banking/transaction-api:v1.4.2
        # DON'T: Copy ports, probes, volumes, and other parameters identical to the base!
        ports:
        - containerPort: 8080 
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080

# ==============================================================================
# CORRECT: Write ONLY the lines to be replaced (Minimalist Overlays).
# Kustomize merges the rest automatically from the base.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transaction-api
spec:
  replicas: 5

If we duplicate the entire manifest structure in patch files, we lose Kustomize’s main benefit. When the base structure in base/ changes (e.g. changing the container port from 8080 to 9000), the production deployment fails to run because the patch file still keeps the old 8080 port overwriting the base file.

2. Storing Plain-Text Sensitive Secrets in Git Repositories #

# k8s/overlays/production/kustomization.yaml
# ANTI-PATTERN: Storing plain-text secret passwords directly in Git repositories
secretGenerator:
  - name: database-secret
    literals:
      - DB_PASSWORD="production-cleartext-password-exposed-here" # DON'T!

Storing secret keys in plain-text in Git is a critical security violation. Anyone with access to the Git repository can see our cluster’s production credentials.

✓ BEST SOLUTION:
  Use external operators like the External Secrets Operator (ESO). 
  Inside Kustomize, we only define the 'ExternalSecret' framework 
  referencing secure secret databases outside the cluster (like AWS Secrets Manager or Vault).

Kustomize Configuration Audit Checklist #

Do a self-audit of your Kustomize directory to make sure production standards are met:

FILE STRUCTURE & HIERARCHY:
  □ The base directory only contains pure manifests free of environment-specific parameters.
  □ Overlay folders don't fully duplicate manifest files; they only contain minimal patches.
  □ The kustomization.yaml file in overlay folders uses the '../../base' relative target path precisely.
  □ All managed resources are registered under the 'resources' array, not imported through ad-hoc external mechanisms.

CONFIGURATION & SECURITY MANAGEMENT:
  □ ConfigMapGenerator is used to manage application configurations to trigger automatic rolling updates when data changes.
  □ SecretGenerator doesn't expose passwords or certificates in plain-text format into Git.
  □ The 'namespace' transformer is used to cleanly separate environment workload isolation in the cluster.
  □ Image tags are declaratively defined under the 'images' block in overlays, not hardcoded in deployment-patch.yaml.

VALIDATION & DEPLOYMENT:
  □ CI/CD pipelines run the 'kubectl diff -k' command before executing apply.
  □ The standalone rendering process has been tested and validated successfully using the 'kubectl kustomize' command.
  □ Global 'commonLabels' are inserted for monitoring standardization and asset ownership tracking.

Summary #

• Template-Free Overlays — Kustomize adopts the template-free principle keeping the original Kubernetes YAML file syntax validity so it’s easy to read and validate.
• Separate Base and Overlays — Make sure base manifests are placed neutrally in the base/ directory, while environment-specific parameters are managed separately in the overlays/ directory.
• Optimize ConfigMap Hash Suffixes — Leverage ConfigMapGenerator to create automatic hash suffixes to trigger Pod rolling updates when configurations are dynamically changed.
• Understand Patching Methods — Use Strategic Merge Patches for smart merging based on merge key attributes, and use JSON Patches for high-precision operations.
• Use Global Transformers — Leverage the namespace, commonLabels, and images features for instant mass metadata modifications on compilation results.
• Avoid Configuration Drift — Avoid direct kubectl edit modification commands post-deployment; all configuration changes must be recorded in the Kustomize Git repository.

← Previous: Helm   Next: kubectl Tips →