Secret #

Storing sensitive credential data — like database passwords, API keys, SSL/TLS certificates, authentication tokens, or SSH keys — requires much stricter handling than regular application configuration. If this data leaks to the public, the consequences can be fatal for business continuity and our users’ data integrity. Kubernetes facilitates storing this sensitive data through a special resource called Secret.

Structurally, Secrets are similar to ConfigMaps because both store data in key-value format. However, Secrets are designed with different handling mechanisms at the cluster system level, like display restrictions on the command line, secure memory placement, and advanced encryption support. Deeply understanding how Secrets work is very important to avoid the false sense of security that often traps beginner developers in production environments.

The Difference Between Secrets and ConfigMaps #

Although they look similar at a glance, Secrets and ConfigMaps have opposite roles and handling characteristics.

Let’s look at the comparative table below to see how Kubernetes deeply distinguishes these two objects:

IMPORTANT: Base64 Is NOT Encryption! Many developers think data inside a Secret object is safe just because it can’t be read directly (appearing as random Base64 characters). Base64 is just an encoding method for converting plain text into safe ASCII character representation, not an encryption algorithm. Anyone with permission to read the Secret YAML manifest can instantly translate it back (decode) to the original text using the base64 --decode command.

Built-in Kubernetes Secret Types #

Kubernetes provides several built-in Secret types, each with special validation rules by the API Server to prevent parameter writing errors:

1. Opaque (Default) #

This is the default Secret type if we don’t specify one explicitly. This type stores free-form key-value data without special validation rules, like database password configuration or Slack webhook tokens.

2. kubernetes.io/tls #

This type is specifically designed to store SSL/TLS certificates along with their private keys. Kubernetes requires data under this type to contain the tls.crt key (for the certificate) and tls.key key (for the private key).

3. kubernetes.io/dockerconfigjson #

This type stores authentication credentials for private container registries (like Docker Hub, AWS ECR, or GCP Artifact Registry). Data must be stored under the .dockerconfigjson key in valid JSON format so the kubelet can use it during the container image pull process.

4. kubernetes.io/service-account-token #

This type is used by the Kubernetes control plane to store authentication tokens used by ServiceAccount objects to interact securely with the API Server.

Secret Creation Methods #

Just like ConfigMaps, we can create Secrets either imperatively or declaratively. However, for declarative manifests, we must convert plain text data into Base64 format first.

Credential Creation Comparison #

Here’s the difference between unsafely creating YAML credentials by storing plain text passwords in the Git repository (anti-pattern) and the correct declarative creation method (solution):

# ANTI-PATTERN: Writing plain text passwords in a YAML manifest then saving it to Git
# ✗ This is a fatal security hole because anyone reading Git can immediately know our password.
apiVersion: v1
kind: Secret
metadata:
  name: db-secret-bad
data:
  password: "my-super-secret-password-123"  # ✗ WRONG: This isn't Base64 format and exposes plain text!
---
# CORRECT: Do the Base64 conversion in the terminal without including a newline
# ✓ Use the -n parameter on the echo command so the '\n' character isn't encoded too.
echo -n "my-super-secret-password-123" | base64

# Terminal output: bXktc3VwZXItc2VjcmV0LXBhc3N3b3JkLTEyMw==

# Then use that value in the declarative YAML manifest
apiVersion: v1
kind: Secret
metadata:
  name: db-secret-good
  namespace: production
type: Opaque
data:
  password: bXktc3VwZXItc2VjcmV0LXBhc3N3b3JkLTEyMw==  # ✓ CORRECT: Valid Base64 value

If we create a Secret imperatively using the kubectl create secret command, the kubectl CLI automatically does the Base64 encoding process for us behind the scenes:

# Creating a Secret imperatively from literals
kubectl create secret generic db-credentials \
  --from-literal=username=app-admin \
  --from-literal=password=S3cur3P@ssw0rd \
  -n production

Using Secrets as Environment Variables #

We can inject Secret data into containers as environment variables so our runtime application can read them. We can inject specific keys using the secretKeyRef parameter or load all entries in bulk with envFrom.

Sensitive Environment Variable Injection Manifest #

Here’s a Deployment manifest example importing database credentials from a Secret object:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-processor
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-processor
  template:
    metadata:
      labels:
        app: payment-processor
    spec:
      containers:
      - name: processor
        image: payment-app:v2.1.0
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASS
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

[!WARNING] Injecting Secrets as environment variables carries a high leak risk. Environment variable values can be exposed through log dump files (crash dumps), internal container process tracing logs, or by other users with permission to run the env command inside the container using kubectl exec.

Using Secrets as Volume Mounts (tmpfs) #

A much safer method for consuming Secret data inside containers is mounting the Secret as physical files in a volume.

The tmpfs (In-Memory RAM) Storage Path for Maximum Security #

When we mount a Secret as a volume, Kubernetes doesn’t write those files to the node’s physical storage media (host SSD/HDD). The kubelet mounts the volume using a RAM memory-based filesystem (tmpfs).

[etcd (Encrypted)] ---> API Server ---> Kubelet ---> Memory RAM (tmpfs Volume) ---> Container Pod
                                                     ^
                                            (No data trace on the host disk)

With this method, as soon as a Pod is destroyed or moved, the Secret data in the worker node’s RAM is immediately erased without leaving any forensic trace on our worker node’s physical storage media.

Secret Volume Mount Implementation Manifest #

Here’s a Deployment manifest example mounting TLS certificates using a volume mount with strict POSIX permissions:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-gateway
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: secure-gateway
  template:
    metadata:
      labels:
        app: secure-gateway
    spec:
      volumes:
      - name: tls-volume
        secret:
          secretName: secure-gateway-tls
          # Set very strict POSIX permissions (0400 = Read-only for the container owner only)
          defaultMode: 256  # Equivalent to octal 0400
      containers:
      - name: gateway-proxy
        image: nginx:1.25.3-alpine
        volumeMounts:
        - name: tls-volume
          mountPath: /etc/nginx/certs
          readOnly: true

Enabling Encryption at Rest on etcd #

By default, all Secret object data is stored in the Kubernetes etcd database as plain text (Base64 plain text). If someone infiltrates the control plane server and directly accesses the etcd database, they can read all our secret credentials. To anticipate this, we must enable Encryption at Rest.

EncryptionConfiguration Setup on the API Server #

We can configure the API Server to automatically encrypt Secret data before writing it to the etcd database by creating an EncryptionConfiguration configuration file on the control plane:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:  # Use the AES-CBC encryption provider with a 32-byte key
      keys:
      - name: key1
        secret: c3VwZXItc2VjcmV0LWtleS1tdXN0LWJlLTMyLWJ5dGVzLXV0Zjg=  # 32-byte Base64 key
  - identity: {}  # Fallback to keep reading old unencrypted Secrets

In managed cloud environments (managed Kubernetes), we don’t have direct access to the API Server configuration. However, cloud providers offer encryption integration options using integrated Key Management Service (KMS) services:

# EKS (AWS): Enable envelope encryption using an AWS KMS Key
aws eks update-cluster-config \
  --name production-cluster \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:ap-southeast-1:123456789012:key/xxxx-xxxx"}}]'

# GKE (Google Cloud): Enable application-level encryption using Cloud KMS
gcloud container clusters update production-cluster \
  --database-encryption-key projects/my-project/locations/asia-southeast1/keyRings/my-ring/cryptoKeys/my-key

Advanced Security Risks Often Ignored #

Using Kubernetes’ built-in Secret objects doesn’t automatically free our application from security threats. There are several silent vulnerabilities often ignored by platform engineers:

1. Leaks Through System Logs: If our application is configured to log all HTTP query arguments or print environment variables during errors (stack traces), Secret values stored in environment variables get written as plain text in log files. These logs are then sent to centralized log aggregation servers (like Elasticsearch/Loki) where non-admin teams can freely read them.
2. Too-Loose RBAC: Default namespace-level RBAC permissions often give developers get and list access rights to all objects in the namespace. Anyone with this access can run kubectl get secrets -o yaml and instantly decode all our production credentials.
3. File Traces in Git Repositories: Teams applying GitOps flows often accidentally add Secret YAML files to Git repositories. Once that file is committed, the data is permanently recorded in Git history even if the file is deleted in the next commit.

RBAC Access Security Comparison #

Here’s the difference between too-permissive default RBAC writing (anti-pattern) and least-privilege RBAC restrictions for Secret reading (solution):

# ANTI-PATTERN: Giving full permissions to all API groups for the application ServiceAccount
# ✗ The application Pod gets read access to all Secrets in that namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role-unsafe
  namespace: production
rules:
- apiGroups: [""]
  resources: ["*"]  # ✗ Gives read permission to all resources including Secrets!
  verbs: ["*"]
---
# CORRECT: Restrict read permissions only to the specific objects needed
# ✓ Only allows reading non-sensitive resources and strictly isolates Secret access.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role-safe
  namespace: production
rules:
- apiGroups: [""]
  resources: ["services", "configmaps"]  # Only non-sensitive resources
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["app-specific-secret"]  # Restrict to only reading this Secret!
  verbs: ["get"]

Secret Management Best Practices in Production #

To ensure our cluster credential security is at an optimal level, we must apply the following guiding principles:

• Restrict RBAC Access with the Least Privilege Principle: Only give Secret read access to authorized operations teams and application Pods that actually need it.
• Use tmpfs Volume Mounts: Whenever possible, mount Secrets as volumes instead of injecting them as environment variables to avoid RAM memory dump log leak risks.
• Use External Secret Managers: Integrate Kubernetes with managed services like AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault. Use operators like External Secrets Operator (ESO) to dynamically sync data from the cloud manager into Kubernetes Secret objects, so developer teams don’t need to write sensitive data in Git repositories.
• Rotate Credentials Periodically: Regularly rotate passwords and API keys (e.g. every 30 or 90 days) to minimize exploitation impact if credentials leak unnoticed.
• Use Git Leak Scanning Tools: Install audit tools like trufflehog or git-secrets in our CI/CD pipeline to early-detect developers accidentally writing or committing Base64 credentials to code repositories.

Summary #

• Base64 isn’t encryption: Always remember Secrets in Kubernetes are only Base64-encoded by default, not encrypted.
• Enable encryption at rest: Make sure etcd-level encryption is enabled in your production cluster, either through cloud KMS providers or local EncryptionConfiguration setup.
• Prefer tmpfs-based volume mounts: Use the volume mount method for consuming Secrets in Pods because data is stored in worker node RAM without leaving physical traces on disks.
• Restrict authorization via RBAC: Use the resourceNames filter on Role/ClusterRole RBAC rules to precisely lock Secret read access per application.
• Use External Secrets Operator (ESO): Apply external secret management integration (AWS/GCP KMS/Vault) to separate credential ownership from cluster GitOps repositories.
• Don’t use sensitive data in ConfigMaps: Always use Secrets for passwords and tokens, and store regular non-sensitive data in ConfigMaps.

← Previous: ConfigMap   Next: Environment Variable Pattern →