Supply Chain Security #

Cloud-native technology security is no longer limited to runtime access restrictions (like RBAC and Pod Security Standards) or network segmentation through NetworkPolicies. One of the most dangerous and continuously evolving attack vectors is software supply chain exploitation. In the Kubernetes ecosystem, the smallest deployment unit is a Pod wrapping a container image. If the container image we deploy is already infected with malware, contains critical security vulnerabilities, or gets replaced by an attacker at the registry level, then all runtime defense mechanisms inside the cluster become useless. Therefore, we must ensure every image running in the Kubernetes cluster can be verified for authenticity, is free of dangerous vulnerabilities, and comes from a trusted build pipeline.

Container Supply Chain Threats #

Before designing a robust defense system, we must first understand how container supply chains can be compromised. These attack vectors vary widely, from the code writing stage on developer local computers, integration processes on CI/CD servers, to image storage in container registries.

Main Attack Vectors #

Container Supply Chain Attack Vectors:

1. Vulnerable or Malicious Base Images
   - Using outdated base images containing critical CVEs (Common Vulnerabilities and Exposures).
   - Using public images deliberately planted with backdoors or cryptocurrency miners by attackers.

2. Third-Party Dependency Vulnerabilities (Dependency Confusion & Typosquatting)
   - Injecting malicious dependencies through public package repositories (npm, PyPI, Maven, etc.) with name similarities.
   - CI/CD pipelines accidentally pull malicious dependencies during application code compilation.

3. Compromised CI/CD Pipelines
   - Attackers hack CI/CD servers (e.g. Jenkins, GitHub Actions runners) and modify build instructions.
   - Injecting malware into application binaries right before the image is wrapped and pushed.

4. Image Tag Manipulation in Registries (Image Spoofing/Overwriting)
   - Without immutability mechanisms, attackers with registry access can overwrite 'latest' or 'v1.0.0' tags with malicious images.
   - Clusters pull the newly modified images without realizing the content changed.

5. Registry Credential Leakage
   - Access tokens or registry passwords stored unsafely (hardcoded) in Git repositories or CI/CD configurations.
   - Attackers use those credentials to upload malicious images directly to our private registry.

To secure this flow, we need a defense-in-depth strategy covering continuous vulnerability scanning, cryptographic signing of build artifacts, transparent software bill of materials generation, and strict policy enforcement before images are allowed to run inside the Kubernetes cluster.

Supply Chain Security Architecture #

To secure the container supply chain from upstream to downstream, we need to integrate various security tools into one unified lifecycle. The flow diagram below shows how code committed by developers is verified, scanned, signed, and validated by the policy engine inside Kubernetes.

flowchart TD
    Developer["Developer Commits Code"] -->|"Git Push"| GitRepo["Git Repository (GitHub/GitLab)"]
    GitRepo -->|"Webhook Trigger"| CIPipeline["CI/CD Pipeline (GitHub Actions/GitLab CI)"]
    
    subgraph BuildScanSign["Build & Hardening Process (CI Node)"]
        direction TB
        BuildImage["Build Container Image (Multi-stage)"] --> ScanTrivy["Scan Vulnerabilities (Trivy)"]
        ScanTrivy -->|"Passes Policy (No Critical CVE)"| SignCosign["Sign Image & Generate Attestation (Cosign)"]
        SignCosign --> GenerateSBOM["Generate SBOM (Syft)"]
    end
    
    CIPipeline --> BuildImage
    GenerateSBOM -->|"Push Image, Signature, & SBOM"| Registry["Secure Private Registry (Harbor/ECR)"]
    
    subgraph K8sAdmission["Kubernetes Cluster Admission Control"]
        direction TB
        DeployReq["kubectl apply / GitOps Sync"] --> K8sAPI["Kube-API Server"]
        K8sAPI -->|"Validating Webhook Request"| Kyverno["Kyverno Policy Engine"]
        Kyverno -->|"Verify Cosign Signature & Registry Policy"| PolicyResult{"Meets the Policy?"}
        PolicyResult -- "Yes" --> PullImage["Allow & Pull the Image to the Kubelet"]
        PolicyResult -- "No" --> RejectDeploy["Reject the Deployment (Blocked)"]
    end
    
    Registry --> PullImage
    K8sAPI -.-> DeployReq

Vulnerability Scanning with Trivy #

The first step in securing the supply chain is ensuring there are no known vulnerabilities inside our container images. Trivy is a fast, reliable vulnerability scanner easily integrated into CI/CD pipelines.

Trivy scans the container operating system (like Debian, Alpine, RedHat) and application dependency packages (like npm, pip, bundler, go.mod), matching them against the global vulnerability database (CVE).

CI/CD Pipeline Integration #

We must not let this scanning process be manual. Scanning must automatically run on every build process in the CI/CD pipeline. If Trivy detects vulnerabilities with high (HIGH) or critical (CRITICAL) severity levels, the build pipeline must automatically fail-fast.

Here’s a comparison between an unsafe Dockerfile configuration (causing many vulnerabilities) and a clean, safe multi-stage Dockerfile approach, followed by the automatic Trivy scanning configuration in GitHub Actions.

# ANTI-PATTERN: The Dockerfile uses a fat base image, runs processes as root,
# and doesn't clean package manager caches, increasing the attack surface and CVE count.
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y python3 python3-pip curl
WORKDIR /app
COPY . /app
RUN pip3 install -r requirements.txt
# Runs the application as the root user by default
CMD ["python3", "main.py"]

# ==============================================================================
# CORRECT: Applying a multi-stage build to eliminate compiler tools,
# using a minimal base image (distroless), and running processes as non-root.
# Stage 1: Build & Dependencies
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# Stage 2: Runtime Image (Distroless)
FROM gcr.io/distroless/python3-debian12:nonroot
WORKDIR /app
# Copy dependencies from the builder stage
COPY --from=builder /root/.local /home/nonroot/.local
COPY . /app
ENV PATH=/home/nonroot/.local/bin:$PATH
# The distroless image automatically runs the process as the nonroot user (ID 65532)
CMD ["main.py"]

After optimizing the Dockerfile, we apply the automatic scanning step in the CI/CD pipeline using the following configuration:

# Example GitHub Actions workflow implementation for scanning with Trivy
name: Security Scan and Build
on:
  push:
    branches: [ "main" ]

jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Build Local Image
        run: |
          docker build -t my-app:${{ github.sha }} .          

      - name: Run Trivy Scan (Fails on CRITICAL vulnerabilities)
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'my-app:${{ github.sha }}'
          format: 'table'
          exit-code: '1' # The build pipeline fails if the criteria are met
          ignore-unfixed: true # Only focus on CVEs that already have official patches
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'

By configuring the --exit-code 1 parameter on Trivy, we force the build process to stop if high-risk vulnerabilities with available fixes (fixed version) are found. This prevents developer teams from pushing vulnerable images to the central repository.

Verifying Image Identity with Cosign #

Having an image free of vulnerabilities doesn’t guarantee security if we can’t prove that the image pulled by the Kubernetes cluster is truly the same image that passed through our trusted CI/CD pipeline. This is where cryptographic signing technology is needed.

Cosign (part of the Sigstore project under the Linux Foundation) simplifies the container signing process. Cosign writes digital signatures directly into the container registry as OCI objects (OCI artifacts) sitting alongside the original container.

The Static Method Using Keypairs #

The conventional method uses a pair of cryptographic keys (public and private keys) created locally or stored in a Key Management Service (KMS) like AWS KMS, GCP KMS, or HashiCorp Vault.

# 1. Generate a signing keypair on the local/admin machine
# This command produces two files: cosign.key (private) and cosign.pub (public)
# We'll be asked to enter a passphrase to secure the private key.
cosign generate-key-pair

# 2. Sign the container image in the build pipeline
# We must provide the private key (cosign.key) and its password via environment variables.
export COSIGN_PASSWORD="super-secret-passphrase"
cosign sign --key cosign.key registry.company.com/apps/secure-api:v1.0.0

# 3. Manually verify the signature before deployment
# The cluster or operators can verify the image's validity using the public key (cosign.pub).
cosign verify --key cosign.pub registry.company.com/apps/secure-api:v1.0.0

The Keyless Signing Method (Recommended) #

In modern environments, managing static private keys in CI/CD pipelines introduces new risks: if the CI/CD pipeline leaks, attackers can steal the private key to sign malicious images.

To solve this, Cosign introduces the Keyless Signing mechanism. This mechanism leverages OpenID Connect (OIDC) integration between the CI/CD pipeline identity provider (like GitHub Actions) and the Sigstore infrastructure (Fulcio as the Certificate Authority and Rekor as the Transparency Log).

flowchart LR
    CI["CI Runner (GitHub Actions)"] -->|"1. Request OIDC Token"| GitHub["GitHub OIDC Provider"]
    GitHub -->|"2. JWT Token"| CI
    CI -->|"3. Send JWT & Temporary Public Key"| Fulcio["Fulcio CA (Sigstore)"]
    Fulcio -->|"4. Verify JWT & Issue an X.509 Certificate"| CI
    CI -->|"5. Sign the Image & Send to the Registry"| Registry["Container Registry"]
    CI -->|"6. Record the Signature Transaction"| Rekor["Rekor Transparency Log"]

When the keyless signing process runs:

1. The build pipeline requests a short-lived OIDC token (JWT) from the platform provider (e.g. GitHub).
2. Cosign sends that token to Fulcio (the Sigstore CA).
3. Fulcio verifies the build pipeline’s identity and issues a very short-lived X.509 certificate (valid for only 10 minutes) bound to the build pipeline identity (e.g. the GitHub Actions workflow URL).
4. Cosign signs the image using a one-time-use (ephemeral key) private key matched to the certificate from Fulcio.
5. The signing details are recorded in Rekor, an append-only (immutable) public transaction log to prevent non-repudiation.
6. The one-time-use private key is immediately destroyed. No static private keys need to be stored or managed in our CI/CD pipeline.

SBOM (Software Bill of Materials) Creation and Management #

To do impact analysis when new vulnerabilities emerge (e.g. zero-day incidents like Log4Shell), we must have full visibility into all libraries and transitive dependencies installed inside the container image. This inventory list is called the Software Bill of Materials (SBOM).

We can use Syft (Anchore’s SBOM generation tool) or Trivy to generate SBOM files in industry-standard formats like CycloneDX or SPDX, then upload them to the registry as part of the container attestation.

The SBOM Creation and Upload Lifecycle #

# 1. Analyze the container and generate an SBOM in CycloneDX JSON format using Syft
syft registry.company.com/apps/secure-api:v1.0.0 -o cyclonedx-json > sbom.json

# 2. Verify the generated SBOM content locally
# Make sure a specific library (e.g. openssl) is in the inventory list
grep -i "openssl" sbom.json

# 3. Upload the SBOM to the Container Registry as an Attestation using Cosign
# The attestation cryptographically binds the SBOM document to the image's SHA256 digest.
cosign attest --key cosign.key \
  --type cyclonedx \
  --predicate sbom.json \
  registry.company.com/apps/secure-api:v1.0.0

# 4. Verify the attestation on the registry side
cosign verify-attestation --key cosign.pub \
  --type cyclonedx \
  registry.company.com/apps/secure-api:v1.0.0

By attaching the SBOM directly to the container in the registry, security teams can centrally scan the vulnerability status of all applications without having to rebuild or unpack container images one by one in the future.

Cluster Policy Enforcement with Kyverno #

Creating cryptographic signatures and vulnerability scans in the CI/CD pipeline isn’t effective if we don’t block non-compliant deployments at the Kubernetes cluster level. We must configure a gateway at the Kubernetes API Server using an admission webhook.

Kyverno is a policy engine specifically designed for Kubernetes. Kyverno lets us validate, mutate, and generate Kubernetes resources using declarative manifests, without needing to write Go code like Open Policy Agent (OPA) Gatekeeper.

The Kyverno Admission Controller Workflow Scenario #

flowchart TD
    Client["kubectl apply / GitOps Sync"] --> APIServer["Kubernetes API Server"]
    
    APIServer -->|"(1) Mutating Webhook: Inject Registry Credentials / Labels"| Kyverno["Kyverno Policy Engine<br/>- Checks the image origin registry<br/>- Downloads the public key from etcd/secret<br/>- Verifies the Cosign signature"]
    APIServer -->|"(2) Validating Webhook: Send the Pod Payload to Kyverno"| Kyverno
    
    Kyverno --> Passed["PASSES THE POLICY<br>Allow the deployment"]
    Kyverno --> Failed["FAILED<br>Reject the deployment with an error message"]

Kyverno Policy Manifest: Trusted Registry Validation #

The following policy ensures the cluster only accepts container images from the company’s internal registry, automatically rejecting images from unknown public registries to prevent shadow deployments.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
  annotations:
    policies.kyverno.io/title: "Restrict Container Image Registries"
    policies.kyverno.io/category: "Supply Chain Security"
    policies.kyverno.io/severity: "High"
    policies.kyverno.io/description: "Ensures all pods only pull images from the organization's official registry domains."
spec:
  validationFailureAction: Enforce # Enforce: reject the deployment, Audit: allow but record violations
  background: true
  rules:
  - name: check-trusted-registries
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Deployment rejected! Image '{{request.object.spec.containers[*].image}}' must use the company official registry at 'registry.company.com/'."
      pattern:
        spec:
          containers:
          # Validation pattern ensuring all containers use the company registry prefix
          - image: "registry.company.com/*"
          # We also validate initContainers if defined in the pod spec
          # using recursive rules or explicit declarations.

Kyverno Policy Manifest: Cosign Signature Verification #

This policy ensures every incoming container image must have a valid digital signature matching the organization’s public key. If an image isn’t signed or is signed with the wrong key, Kyverno blocks the Pod from running.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-image-signatures
  annotations:
    policies.kyverno.io/title: "Verify Cosign Image Signatures"
    policies.kyverno.io/category: "Supply Chain Security"
    policies.kyverno.io/severity: "Critical"
spec:
  validationFailureAction: Enforce
  background: false # Evaluates in real-time when admission requests arrive
  rules:
  - name: verify-cosign-signature
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - imageReferences:
      - "registry.company.com/apps/*"
      attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE9l/S02C3m4J2bA1w8s9dY1wZ
              U8uJ2K9kXj9W6+o/5g43209FASDF09asfFASKL12309ASDFK12309ASDFK12
              3u3X+N9w5xP8C5p5C0B7wY==
              -----END PUBLIC KEY-----              

By applying this policy at the cluster level, we close the gap for anyone (including internal teams with cluster administrator access) to run raw images from outside the verified CI/CD pipeline.

Container Registry Hardening & Pull Secrets #

The container registry is the heart of software distribution in Kubernetes. We must treat the registry as a critical system requiring security hardening.

Immutability Tags #

One of the most common mistakes is allowing image tag rewriting (tag mutability). For example, if we allow the v1.0.0 tag to be updated with a new image, we lose consistency guarantees.

Tag Mutability vs Immutability Patterns:

[MUTABLE TAGS - HIGH RISK]
Developer Build #1 ──> Push app:v1.0.0 (SHA: aaa) ──> Kubernetes Pull (Run SHA: aaa)
Developer Build #2 ──> Push app:v1.0.0 (SHA: bbb) ──> Kubernetes Pull (Run SHA: bbb)
✗ The cluster runs two different code versions with the same tag label.
✗ Audit logging becomes chaotic because tags don't represent unique code.

[IMMUTABLE TAGS - SAFE]
Developer Build #1 ──> Push app:v1.0.0 (SHA: aaa) ──> Success
Developer Build #2 ──> Push app:v1.0.0 (SHA: bbb) ──> REJECTED BY THE REGISTRY (Tag already exists)
✓ We're forced to raise the version (e.g. app:v1.0.1) for every change.
✓ Every cluster deployment is deterministic and safely rollback-able.

We must enable the Immutable Tags option on the container registry we use (like Harbor, AWS ECR, GCP Artifact Registry, or Azure ACR).

Authentication Using Image Pull Secrets #

Running a private registry means the Kubernetes cluster needs credentials to pull images. We must avoid using global credentials with write access. Use dedicated accounts with read-only access.

Here’s a comparison example of handling image pull credentials:

# ANTI-PATTERN: Manually creating docker-registry Secrets in every namespace randomly,
# or attaching credentials directly to deployment manifests. This makes credential rotation difficult.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: insecure-app
  namespace: development
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: web
        image: registry.company.com/apps/web:v1.0.0
      # No automatic secret handling, developers are used to copy-pasting secrets
      imagePullSecrets:
      - name: manual-registry-secret

# ==============================================================================
# CORRECT: Binding Registry Credentials to the Namespace's Default ServiceAccount.
# Every Pod running with this ServiceAccount automatically inherits image pull access.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: secure-service-account
  namespace: production
# Binds the pull secret at the ServiceAccount level
imagePullSecrets:
- name: global-read-only-registry-secret
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      serviceAccountName: secure-service-account # Uses the trusted ServiceAccount
      containers:
      - name: web
        image: registry.company.com/apps/web:v1.0.0
        resources:
          limits:
            cpu: "500m"
            memory: "512Mi"
          requests:
            cpu: "200m"
            memory: "256Mi"

By integrating the pull secret into the ServiceAccount, developer teams no longer need to write imagePullSecrets blocks in every application deployment file. This reduces human configuration errors and simplifies registry token lifecycle management.

Minimizing the Attack Surface with Distroless and Multi-Stage Builds #

Every line of code and utility inside a container image that isn’t used by the application at runtime is an additional security burden. Tools like curl, wget, apt, npm, or sh/bash shells inside runtime images are weapons for attackers to do lateral exploitation if our application gets breached.

Base Image Characteristics Comparison #

Why Is Distroless Safer? #

Distroless containers only contain our application and its minimal runtime dependencies. There’s no full operating system, package manager, shell, or other standard Linux utilities inside the distroless image.

If an attacker finds a Remote Code Execution (RCE) vulnerability in our Python application running on a distroless container:

• The attacker can’t run shell commands because /bin/sh or /bin/bash don’t exist.
• The attacker can’t download backdoor scripts using curl or wget because those tools aren’t installed.
• The attacker can’t install new penetration tools because there’s no apt or pip.

This significantly limits post-exploitation movement space at the container kernel level.

Supply Chain Security Anti-Patterns #

Let’s explore several common mistakes (anti-patterns) DevOps teams often make in managing the container supply chain and the right solutions to fix them.

1. Using the :latest Tag in Production Environments #

# ANTI-PATTERN: Pulling base images using the dynamic 'latest' tag.
# Every time the pipeline runs, the base image can change unpredictably.
FROM node:latest
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "app.js"]

# ==============================================================================
# CORRECT: Pinning the base image version using a precise specific tag
# and identifying the SHA256 digest to guarantee absolute determinism.
FROM node:20.11.0-alpine3.19@sha256:7bc5be6704044...
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["node", "app.js"]

2. Ignoring Signature Verification in the Cluster Environment #

Many teams sign images in the CI/CD pipeline but don’t enable an admission controller to verify those signatures in Kubernetes. As a result, the protection is only administrative outside the cluster, while the cluster remains vulnerable to direct foreign image execution.

3. Running Scans Only Once at Build Time #

New vulnerabilities (zero-days) are discovered every day. A container image declared clean of CVEs when built two months ago may now contain critical-risk vulnerabilities. We must do scheduled scans of images currently running inside the cluster, using tools like the Trivy Operator, which actively updates the security status of all running Pods.

Supply Chain Security Checklist #

Before moving to the production stage, make sure all items in the following checklist have been implemented and their functionality verified:

BUILD STAGE (CI/CD PIPELINE):
  □ The Dockerfile uses a multi-stage build approach to separate build tools.
  □ Base images use minimal versions (Distroless or Alpine) with specific tags (SHA256 digests).
  □ Automatic Trivy scanning is enabled on every commit/pull request.
  □ The CI/CD pipeline is configured to fail (exit code 1) if CRITICAL or HIGH vulnerabilities are detected.
  □ Cosign is integrated to cryptographically sign images after the build finishes.
  □ SBOM document files (CycloneDX JSON format) are generated and uploaded as attestations to the registry.

STORAGE STAGE (REGISTRY):
  □ The "Immutable Tags" policy is enabled on the main container registry.
  □ Registry access credential rotation is applied automatically and periodically.
  □ Write access to the registry is restricted to official CI/CD pipeline service accounts only.
  □ The registry's built-in vulnerability scanning (Harbor / ECR / ACR scan-on-push) is enabled.

RUNTIME STAGE (KUBERNETES CLUSTER):
  □ The Kyverno Policy Engine is applied to supervise and filter every created pod.
  □ Kyverno rules rejecting container images from untrusted external registries are installed.
  □ Kyverno rules validating Cosign signatures against the official public key are enabled.
  □ Image Pull Secret credentials are bound to the default ServiceAccount instead of being written in every deployment.
  □ The Trivy Operator runs inside the cluster to monitor running image vulnerabilities in real-time.

Summary #

• Supply Chain Security Is the Master Key — All Kubernetes cluster runtime protections collapse if the initial container image we deploy was already compromised by an attacker at the pipeline or registry level.
• Trivy Stops Vulnerable Images Early — Vulnerability scanning inside the CI/CD pipeline with Trivy proactively stops uploading images with critical security holes (fail-fast).
• Cosign Guarantees Code Authenticity — Cosign’s cryptographic signatures ensure the image running in the cluster is truly the official compilation result from the trusted build pipeline, not an impostor image.
• Kyverno Becomes the Cluster Gatekeeper — The Kyverno policy engine acts as a validating admission controller automatically blocking any image lacking valid signatures or coming from external registries.
• Minimal Base Images Narrow the Attack Surface — Using distroless images without shells, package managers, and external tools prevents attackers from doing lateral exploitation if our application gets breached.
• SBOM Facilitates Quick Audits — Software component inventories (SBOMs) uploaded to the registry make it easy to track affected dependencies when new public zero-day vulnerabilities are discovered.

← Previous: Network Security   Next: Audit Logging →