Secret Management Best Practice #

Managing sensitive data (secrets) inside a production Kubernetes cluster isn’t just about creating Secret objects, storing them in the system, and hoping everything is safe. In the cluster’s default configuration, Secret objects are only paired with Base64 encoding that anyone can instantly decode. Additionally, the etcd database stores that data as plain text by default, there’s no built-in automatic credential rotation mechanism, and there’s no detailed audit log recording to detect who has accessed our credentials.

To build a truly secure cluster infrastructure, we must apply the defense-in-depth principle. We must protect sensitive data starting from the physical storage level, the API Server authorization level, all the way to the secret’s lifecycle level inside application code. This article deeply discusses best practices for securing, auditing, synchronizing, and rotating Secrets in our production Kubernetes cluster environments.

The Five Layers of Secret Security #

When designing Secret security, we must not rely on only one defense mechanism. If one defense layer gets breached (e.g. a hacker successfully infiltrates a worker node), the other defense layers must still be able to protect our main credentials.

Let’s look at the five integrated security layer architecture below that we must apply in production clusters:

flowchart TD
    subgraph Layer1["Layer 1: Encryption at Rest (Storage)"]
        etcd["etcd Database (KMS Encrypted)"]
    end

    subgraph Layer2["Layer 2: Encryption in Transit"]
        etcd -->|"Self TLS Encryption"| APIServer["kube-apiserver"]
    end

    subgraph Layer3["Layer 3: Access Control (Authorization)"]
        APIServer -->|"RBAC Least Privilege"| Client["kubectl / Pod ServiceAccount"]
    end

    subgraph Layer4["Layer 4: Audit & Monitoring (Tracking)"]
        APIServer -->|"Audit Logging Policy"| AuditLog["SIEM / Elasticsearch logs"]
    end

    subgraph Layer5["Layer 5: Lifecycle Management (Rotation)"]
        Client -->|"Dynamic Automatic Rotation"| App["Application Container Pod"]
    end

Advanced Encryption at Rest Configuration #

By default, the data stored by the API Server into etcd isn’t encrypted. If a hacker successfully steals etcd backup files or breaks into the control plane, all our production passwords get exposed. Therefore, enabling Encryption at Rest is the first mandatory step in production clusters.

Example EncryptionConfiguration Manifest on Self-Managed Clusters #

On self-managed clusters (e.g. using kubeadm), we must create the /etc/kubernetes/enc.yaml configuration file on the control plane and reference it in the API Server configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:  # Use the AES-CBC provider for fast kernel encryption performance
      keys:
      - name: key1
        secret: bXktc3VwZXItc2VjcmV0LWtleS1tdXN0LWJlLTMyLWJ5dGVzLXV0Zjg=  # 32-byte Base64 key
  - identity: {}  # Fallback for reading old unencrypted Secrets

The Data Rewrite Procedure After Enabling Encryption #

[!WARNING] When we enable encryption on the API Server, Kubernetes doesn’t automatically encrypt Secrets already in the etcd database. This encryption policy only applies to new Secret data written after the configuration is enabled.

To retroactively encrypt all old Secrets already stored in etcd, we must run a bulk read-and-rewrite command:

# Command to force rewriting all Secrets in all cluster namespaces
# This replace action makes the API Server rewrite data to etcd using the active encryption key
kubectl get secrets -A -o json | kubectl replace -f -

Restricting Access with Granular RBAC #

A common mistake in DevOps teams is granting broad Secret read permissions on a namespace. get and list access rights on Secret objects must not be given to non-admin developers or application ServiceAccounts that don’t need them.

RBAC Authorization Manifest Comparison #

Here’s the difference between too-loose, dangerous access grants (anti-pattern) and specific access restrictions using the resourceNames filter (solution):

# ANTI-PATTERN: Giving full access rights to all Secrets in the production namespace
# ✗ The application Pod can read payment API keys and database passwords belonging to other apps
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: generic-app-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]  # ✗ Too permissive, gives access to all Secrets
---
# CORRECT: Restrict access rights to only get on specific Secret names
# ✓ Safe, limits verbs to only 'get' (no list) and explicitly locks the Secret name
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: specific-db-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["postgres-db-creds"]  # ✓ Locks the Secret name specifically
  verbs: ["get"]  # ✓ No list/watch to prevent data crawling

To audit and make sure our application ServiceAccount access rights are properly isolated, use the following diagnostic command utility:

# Test whether a specific ServiceAccount has permission to read Secrets in the production namespace
kubectl auth can-i get secrets -n production \
  --as=system:serviceaccount:production:my-app-sa

# Expected output: "no"

External Secret Synchronization: Integration with Cloud Secret Managers #

Storing Base64 Secret values directly in YAML manifests committed to Git repositories is a fatal security violation in GitOps principles. To solve this, we must use an external secret manager like AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault.

The best tool for syncing data from a cloud manager into a Kubernetes cluster is External Secrets Operator (ESO).

Example External Secrets Operator (ESO) Implementation with AWS #

We can define a SecretStore object to connect the cluster to AWS Secrets Manager using ServiceAccount authentication bound to an IAM role (IRSA):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager-store
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        jwt:
          # References the local ServiceAccount bound to the AWS IAM Role
          serviceAccountRef:
            name: eso-aws-auth-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sync-db-credentials
  namespace: production
spec:
  refreshInterval: "1h"  # Sync data from AWS every 1 hour
  secretStoreRef:
    name: aws-secrets-manager-store
    kind: SecretStore
  target:
    name: local-db-secret  # The name of the Kubernetes Secret object auto-created
    creationPolicy: Owner
  data:
  - secretKey: db-password  # The destination key in the Kubernetes Secret
    remoteRef:
      key: production/rds/postgres  # The key name in AWS Secrets Manager
      property: password  # The specific JSON property in AWS

With this pattern, our Git repository only stores safe ExternalSecret manifests (no sensitive values). The actual secret values stay in the cloud provider and get automatically synced to the cluster.

Alternative: HashiCorp Vault with Vault Agent Injector #

If we use HashiCorp Vault as a centralized secret manager server, we have the option to project secrets directly into the Pod’s filesystem without ever storing them as Secret objects in the cluster’s etcd database. This approach uses the Vault Agent Injector, which works based on container annotations.

When the API Server detects specific Vault annotations on a Pod, it injects an init container and a Vault Agent sidecar container that automatically fetch secrets from Vault and write them to the shared memory directory /vault/secrets/.

Here’s an example Deployment manifest with Vault Agent annotation configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-gateway
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: billing-gateway
  template:
    metadata:
      labels:
        app: billing-gateway
      annotations:
        # Enables Vault agent injection
        vault.hashicorp.com/agent-inject: "true"
        # Determines the allowed authentication role in Vault
        vault.hashicorp.com/role: "billing-app-role"
        # Determines the secret path in Vault and the destination file name inside the Pod
        vault.hashicorp.com/agent-inject-secret-db-config: "secret/data/production/db"
        # Writes a custom template composing credentials into an env file format
        vault.hashicorp.com/agent-inject-template-db-config: |
          {{- with secret "secret/data/production/db" -}}
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASSWORD="{{ .Data.data.password }}"
          export DB_URL="postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres-master:5432/billingdb"
          {{- end }}          
    spec:
      containers:
      - name: gateway-app
        image: billing-app:v3.0.0
        command: ["/bin/sh", "-c", "source /vault/secrets/db-config && ./run-app"]

With the Vault Agent Injector pattern:

  • Secrets only exist in the running container’s RAM memory.
  • Secret data is never stored in the Kubernetes etcd database, eliminating data leak risks if etcd backups are compromised.
  • Secret rotation in Vault immediately rewrites the /vault/secrets/db-config file in real-time inside the container.

Automatic Credential Rotation Strategies Without Downtime #

Keeping one database password unchanged for years in production is a high security risk. We must rotate credentials periodically. However, manual rotation often triggers application downtime because of the time gap between changing the password in the database and updating it in the application.

To rotate without downtime, we must adopt the Double-Credentials Pattern.

DOWNTIME-FREE CREDENTIAL ROTATION FLOW:

  Step 1: The database is configured to accept two active credentials at once:
             - Credential A (Actively used by old Pods)
             - Credential B (Newly created for the transition process)
  
  Step 2: Update the Secret in the External Secret Manager with Credential B
  Step 3: ESO detects the change and syncs Credential B to the cluster
  Step 4: Trigger a Deployment rolling update:
             - New Pods start using Credential B
             - Old Pods keep running with Credential A
  
  Step 5: After all new Pods are healthy, remove Credential A from the database

Fast Rotation Synchronization Configuration #

We can set the refreshInterval parameter on the ExternalSecret manifest to a faster duration to speed up credential change detection:

spec:
  refreshInterval: "15m"  # Sync the new version within max 15 minutes

Safe GitOps Patterns for Secrets (Secrets-as-Code) #

For organizations applying full GitOps where all cluster configuration must be represented in Git, there are three popular options for managing Secrets without leaking their data:

1. Sealed Secrets (Bitnami) #

Sealed Secrets uses asymmetric cryptography to secure data. We encrypt the Secret object using the kubeseal CLI tool on a local computer, leveraging the public key obtained from the Sealed Secrets controller running inside the cluster.

# Converting a standard Secret into an encrypted SealedSecret
# This encrypted result is very safe to commit to public Git repositories
kubeseal --controller-name=sealed-secrets --format=yaml < db-secret.yaml > db-sealedsecret.yaml

The encryption result is a custom SealedSecret YAML manifest file that can be committed to Git:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: production-db-secret
  namespace: production
spec:
  encryptedData:
    # Only the Sealed Secrets controller inside the cluster has the private key to decrypt this data
    database-password: AgBy8BbKJk9p8Y8SbOzU23hY7gHGtT2...

2. Mozilla SOPS (Secrets Operations) #

SOPS is an encrypted text file editor supporting YAML, JSON, ENV, INI, and BIN formats. SOPS integrates the encryption process with cloud KMS (like AWS KMS, GCP KMS, Azure Key Vault, or HashiCorp Vault) or local GPG.

SOPS’s advantage is that it only encrypts the data values, while the key structure and Kubernetes manifest metadata stay as plain text so we can easily read the manifest framework in Git.

# Encrypting a secret file using an AWS KMS key
sops --encrypt --kms "arn:aws:kms:ap-southeast-1:123456789012:key/xxxx" secret.yaml > secret.enc.yaml

Audit Logging Policy and Secret Access Monitoring #

Secret security is incomplete without access tracking (audit trail). We must know who (users or ServiceAccounts) has made API calls to read our cluster Secrets to early-detect suspicious activity.

We can create an audit filtering rule on the API Server by writing the /etc/kubernetes/audit-policy.yaml file:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# 1. Record all write requests (create, update, delete) on Secrets at the RequestResponse level
- level: RequestResponse
  resources:
  - group: "" # Core API Group
    resources: ["secrets"]
  verbs: ["create", "update", "patch", "delete"]

# 2. Record read access (get, list, watch) by users at the Metadata level for log size efficiency
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["get", "list", "watch"]

# 3. Ignore Secret query logging by internal kubelet systems to reduce log noise
- level: None
  users: ["system:node", "system:kube-scheduler", "system:kube-controller-manager"]
  resources:
  - group: ""
    resources: ["secrets"]

These audit logs should then be sent to a centralized log repository or SIEM system (like Splunk or Elasticsearch) to trigger alerts if mass list secrets queries are detected outside working hours.

Summary #

  • Enable etcd encryption at rest: Don’t let the etcd database store secrets in plain text Base64 format; enable encryption using a KMS provider.
  • Rewrite after enabling encryption: Run the bulk replace command to force old Secret data to be rewritten to etcd in encrypted format.
  • Apply granular RBAC: Use the resourceNames parameter to restrict Secret read access only to specific authorized Pods.
  • Use External Secrets Operator: Avoid storing Secret value representations in Git; automatically sync secrets from AWS/GCP/Vault using ESO.
  • Adopt the double-credentials rotation pattern: Rotate database credentials gradually using two active credentials to avoid application downtime during transitions.
  • Configure the API Server Audit Policy: Record every read-write access to Secret objects in audit logs to detect potential data leaks.

← Previous: Environment Variable Pattern   Next: External Secret Manager →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact