External Secret Manager #

Kubernetes’ built-in secret data storage has serious operational limitations for enterprise-scale infrastructure needs. In large-scale cluster environments, we’re expected to manage hundreds of credentials spread across dozens of namespaces and several different clusters. If we only rely on native Kubernetes Secrets: we lose centralized audit tracking ability, have no automatic credential rotation feature, no secret versioning history, and are prone to human errors if sensitive data gets committed into GitOps repositories.

To close this security gap, modern industry adopts an integration pattern with External Secret Managers — like AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, or HashiCorp Vault. These external systems are specifically designed with hardware cryptography (HSM) features, super-strict audit logs, automatic credential rotation, and separation of duties. This article fully dissects external secret manager architecture, implementation using External Secrets Operator (ESO), and a guide to choosing the right solution for our production clusters.

The Need for External Secret Integration #

Switching from native Kubernetes Secrets to managed external solutions provides significant security improvements for the production ecosystem.

Let’s look at the comparative table below to see the differences in depth:

External Secrets Operator (ESO) Architecture and How It Works #

To connect data sources from external secret managers into Kubernetes, we need an active synchronization agent. The current industry de facto standard is using External Secrets Operator (ESO). ESO is a custom controller-based Kubernetes operator that periodically watches secret changes in the cloud provider, then securely rewrites their values into local Kubernetes Secret objects.

Let’s study the dynamic data synchronization flow performed by ESO inside the cluster through the following diagram:

flowchart TD
    subgraph ExternalProvider["Cloud Secret Provider"]
        AWS_SM["AWS Secrets Manager / GCP SM / Vault"]
    end

    subgraph K8sCluster["Kubernetes Cluster Boundary"]
        ESO["External Secrets Operator Controller"]
        SecretStore["SecretStore / ClusterSecretStore"]
        ExtSecret["ExternalSecret Resource"]
        K8sSecret["Kubernetes Secret (Target)"]
        Pod["Application Container Pod"]
        
        SecretStore -.->|"Provides IAM / JWT Authentication"| AWS_SM
        ExtSecret -->|"References the Authentication Chain"| SecretStore
        ESO -->|"Watch for Changes"| ExtSecret
        ESO -->|"1. Pull New Secret Data"| AWS_SM
        ESO -->|"2. Write & Sync Data"| K8sSecret
        K8sSecret -->|"3. Consumed at Runtime"| Pod
    end

ESO uses two main Custom Resource Definitions (CRDs):

1. SecretStore / ClusterSecretStore: Defines the connection and authentication method to the cloud provider. SecretStore is bound to a specific namespace, while ClusterSecretStore is global and usable by all namespaces in the cluster.
2. ExternalSecret: Defines the key mapping of which secrets to pull from the cloud provider and the target Kubernetes Secret object name to generate.

Deep Integration with AWS Secrets Manager #

For clusters running on AWS (EKS), we can use the IAM Roles for Service Accounts (IRSA) authentication method to connect ESO to AWS Secrets Manager without storing static credentials (AWS Access Key ID / Secret Access Key) inside the cluster.

AWS Synchronization Authentication Comparison #

Here’s the difference between configuring the connection using dangerous static keys (anti-pattern) and dynamic IAM Role-based authentication (solution):

# ANTI-PATTERN: Using static AWS Access Keys inside the manifest
# ✗ Very dangerous because AWS admin access keys get exposed inside the cluster
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-store-unsafe
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          awsAccessKeyID:
            name: static-aws-creds  # Static keys at risk of leaking
            key: access_key
          awsSecretAccessKey:
            name: static-aws-creds
            key: secret_key
---
# CORRECT: Apply IRSA (IAM Roles for Service Accounts) authentication
# ✓ No static keys, access is authorized through dynamic OIDC token federation
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        jwt:
          # Uses the ServiceAccount dynamic token bound to the AWS IAM Role
          serviceAccountRef:
            name: external-secrets-operator-sa
            namespace: external-secrets

After the ClusterSecretStore is created, we can create an ExternalSecret object to map secret database parameters from AWS Secrets Manager into the application namespace:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: billing-db-secret
  namespace: billing
spec:
  refreshInterval: "1h"  # Run a sync query every 1 hour
  secretStoreRef:
    name: aws-secrets-manager-store
    kind: ClusterSecretStore
  target:
    name: billing-postgres-creds  # The local Kubernetes Secret name to create
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: production/billing/db
      property: db_username
  - secretKey: password
    remoteRef:
      key: production/billing/db
      property: db_password

Deep Integration with HashiCorp Vault #

HashiCorp Vault is one of the most popular secret managers because of its mature multi-cloud architecture support. ESO’s integration with Vault usually uses the Kubernetes Auth Method authentication, where Vault verifies the Kubernetes ServiceAccount’s JWT token.

SecretStore Configuration Manifest for HashiCorp Vault #

Here’s an example SecretStore manifest configured to connect to an external Vault cluster:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend-store
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.internal.example.com:8200"
      path: "secret"  # The KV secrets engine mount path in Vault
      version: "v2"   # Uses KV version 2 (with versioning)
      auth:
        kubernetes:
          mountPath: "kubernetes"  # The auth method path in Vault
          role: "production-app-role"  # The role matched in Vault
          serviceAccountRef:
            name: app-service-account  # The Pod's local ServiceAccount

Standout Feature: Dynamic Secrets Engine #

One of HashiCorp Vault’s biggest advantages over cloud-native secret managers is its ability to generate dynamic credentials (Dynamic Secrets).

When a container requests database access through Vault, Vault doesn’t return a static database admin password. Vault automatically creates a new database user with a random password on the target PostgreSQL server, grants minimal access rights, and sets an active time limit (TTL / Time to Live, e.g. 1 hour). Once the TTL ends, Vault automatically deletes that user from the database server.

# ESO manifest for fetching Dynamic Credentials from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dynamic-db-creds
  namespace: production
spec:
  refreshInterval: "45m"  # Pull new credentials before the 1-hour TTL ends
  secretStoreRef:
    name: vault-backend-store
    kind: SecretStore
  target:
    name: dynamic-db-secret
  data:
  - secretKey: db-username
    remoteRef:
      key: database/creds/read-only-role  # The dynamic database engine path
      property: username
  - secretKey: db-password
    remoteRef:
      key: database/creds/read-only-role
      property: password

This pattern is very robust at suppressing hacking risks, because no permanent database passwords are stored in our cluster configuration files.

Integration with Google Cloud (GCP) Secret Manager #

In Google Kubernetes Engine (GKE) environments, we integrate synchronization using the Workload Identity method. Workload Identity maps a Kubernetes ServiceAccount directly to a Google Service Account (GSA) at the GCP IAM level.

1. ClusterSecretStore for GCP Secret Manager #

Here’s the GCP Secret Manager integration configuration manifest using Workload Identity:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-store
spec:
  provider:
    gcpsm:
      projectID: "my-gcp-production-project"
      auth:
        workloadIdentity:
          clusterLocation: asia-southeast1
          clusterName: gke-prod-cluster
          serviceAccountRef:
            name: eso-gcp-sync-sa
            namespace: external-secrets

2. ExternalSecret for GCP Secret Manager #

After the ClusterSecretStore is active, we can pull secret data from GCP Secret Manager using the following manifest:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gcp-db-secret
  namespace: production
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: gcp-db-secret  # The local Kubernetes Secret name generated
  data:
  - secretKey: db-password
    remoteRef:
      key: production-db-password  # The secret name in GCP Secret Manager
      version: "latest"            # The secret version (can specify a specific version number)

Integration with Azure Key Vault #

For organizations operating infrastructure on Microsoft Azure (AKS), we can sync sensitive data from Azure Key Vault using the Azure AD Workload Identity authentication method.

1. SecretStore for Azure Key Vault #

Here’s a SecretStore manifest configured to connect our namespace to the Azure Key Vault service:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-key-vault-store
  namespace: production
spec:
  provider:
    azurekv:
      tenantId: "12345678-abcd-1234-abcd-1234567890ab"  # Your Azure Directory Tenant ID
      vaultUrl: "https://my-prod-key-vault.vault.azure.net"  # Your Azure Key Vault URL
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: eso-azure-sync-sa  # Local ServiceAccount bound to an Azure Managed Identity

2. ExternalSecret for Azure Key Vault #

Use the following manifest to pull secrets or keys from Azure Key Vault and turn them into Kubernetes Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: azure-app-secret
  namespace: production
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: azure-key-vault-store
    kind: SecretStore
  target:
    name: local-azure-secret
  data:
  - secretKey: api-token
    remoteRef:
      key: production-api-token  # The secret name in Azure Key Vault

Choosing the Best Solution Based on Needs #

Each external secret manager has different performance, integration, and cost characteristics. Choosing a solution that doesn’t fit the infrastructure needs can unnecessarily increase operational complexity.

Let’s study the decision tree below to determine the best solution choice:

flowchart TD
    Start{"Start Choice Evaluation"} --> IsMultiCloud{"Are you using Multi-Cloud / On-Premise?"}
    
    IsMultiCloud -- "Yes" --> UseVault["Primary Choice: HashiCorp Vault"]
    
    IsMultiCloud -- "No" --> CloudProvider{"Where does the Kubernetes Cluster run?"}
    CloudProvider -- "AWS (EKS)" --> UseAWS["Choice: AWS Secrets Manager"]
    CloudProvider -- "GCP (GKE)" --> UseGCP["Choice: GCP Secret Manager"]
    CloudProvider -- "Azure (AKS)" --> UseAzure["Choice: Azure Key Vault"]
    
    UseVault --> FeaturesVault["Advantages: Dynamic Secrets, PKI engine, Multi-Cloud"]
    UseAWS --> FeaturesAWS["Advantages: RDS auto-rotation, native KMS integration"]
    UseGCP --> FeaturesGCP["Advantages: Workload Identity integration, cost-effective"]

Here’s a technical characteristic comparison summary to help the platform engineer team’s decision-making process:

1. HashiCorp Vault:
   • Advantages: Very portable, supports dynamic secrets, automatic PKI management, very rich RBAC control.
   • Disadvantages: Requires high operational costs for installation, cluster backup maintenance, and security policy team operations.
2. AWS Secrets Manager:
   • Advantages: Fully managed, built-in integration with RDS database automatic rotation systems, native KMS authorization.
   • Disadvantages: AWS vendor lock-in, API calls cost can get quite expensive if the ESO refreshInterval is set too aggressively.
3. GCP Secret Manager:
   • Advantages: Very simple, password-less authentication using Workload Identity, Cloud Logging audit log integration.
   • Disadvantages: Automatic rotation features aren’t as rich as AWS, GCP vendor lock-in.

Summary #

• Use ESO as the synchronization standard: Apply External Secrets Operator to transparently and consistently sync sensitive data from outside into cluster Secret objects.
• Must use IRSA/Workload Identity authentication: Avoid writing static access keys in YAML manifests; use dynamic JWT tokens authorized by the cloud provider’s OIDC.
• Leverage dynamic secrets for database protection: Use Vault’s Dynamic Secrets Engine feature to minimize permanent database credential theft risks.
• Choose solutions based on the cluster environment: Use AWS Secrets Manager if fully running on EKS, GCP Secret Manager on GKE, and HashiCorp Vault for multi-cloud needs.
• Set the refreshInterval parameter wisely: Don’t set sync times too often (e.g. every 5 seconds) on cloud-managed secret managers to avoid ballooning cloud API call bills.
• Maintain the separation of duties principle: Developer teams may only create ExternalSecret manifests, while ClusterSecretStore connection configuration is managed specifically by the platform security team.

← Previous: Secret Management Best Practice   Next: ConfigMap vs Secret →