GitOps #

In the traditional infrastructure operations paradigm, Continuous Delivery (CD) pipelines are managed using a Push-Based approach. Integration servers (like Jenkins, GitLab CI, or GitHub Actions runners) hold cluster admin credential files (kubeconfig), execute Bash scripts, and directly run kubectl apply commands from outside the cluster. This approach inserts a serious security hole because cluster admin credentials are scattered outside the cluster’s defense boundary, and it makes detecting configuration deviations hard if someone manually changes object states inside the cluster.

GitOps arrives as an operational method evolution solving those weaknesses. GitOps defines the entire desired state of the system — from pure Kubernetes manifests, Helm values, Kustomize configuration, to ingress parameters — into a Git repository acting as the Single Source of Truth. Using a Pull-Based approach, a controller agent (GitOps operator) running internally inside the Kubernetes cluster periodically monitors that Git repository. If a difference between Git and the cluster is detected (drift), the agent actively re-aligns the cluster state to be identical to what’s written in the Git repository.

The CD Methodology Spectrum: Push-Based vs Pull-Based (GitOps) #

To deeply understand GitOps’ advantages, we must compare the delivery architecture differences between the traditional Push-Based model and the Pull-Based GitOps model:

flowchart TD
    subgraph Push["PUSH-BASED MODEL (TRADITIONAL)"]
        Dev1["Developer"] -->|"Push Code"| CI1["CI Pipeline"]
        CI1 --> CD1["CD Runner (kubectl apply)"]
        CD1 --> Cluster1["Cluster"]
        Cred1["Holds Kubeconfig Credentials\n*Potential Security Hole*"] -.-> CD1
    end

    subgraph Pull["PULL-BASED MODEL (GITOPS)"]
        Dev2["Developer"] -->|"Push Code"| CI2["CI Pipeline"]
        CI2 -->|"Push Image & Update Git Config"| GitRepo2["Git Config Repo"]
        GitRepo2 -. "Watch & Pull Manifests" .-> Operator2["GitOps Operator (ArgoCD / FluxCD inside the Cluster)"]
        Operator2 -->|"In-Cluster Apply"| Cluster2["Cluster"]
    end

Push-Based Model Weaknesses: #

1. Credential Exposure: External CI/CD servers must store high-privilege production cluster authentication tokens (ClusterAdmin). If that CI/CD server gets hacked, attackers immediately fully control our Kubernetes cluster.
2. Configuration Drift: If an operator directly modifies the production cluster using kubectl edit commands, the external CI/CD server never knows about it. The cluster’s actual state deviates from the code in Git.
3. Weak Audit Trails: There’s no centralized record showing who modified the cluster state if changes are done ad-hoc directly inside containers.

Pull-Based Model (GitOps) Advantages: #

1. Zero Credential Exposure: External CI/CD teams only have access to commit/push to the Git repository. Production cluster credentials stay safely isolated within the cluster boundary.
2. Self-Healing: GitOps operators continuously monitor deviations. If an unauthorized manual change happens in the cluster, the operator overwrites that change and returns the state to what’s registered in the Git repository.
3. Instant Disaster Recovery: If the entire production cluster suffers total failure, we just build a new empty cluster, install the GitOps operator, and point it at the Git repository. Within minutes, the operator rebuilds the entire application state precisely like the pre-crash condition.

Dissecting the ArgoCD and FluxCD Architectures #

In the Kubernetes ecosystem, there are two dominant GitOps operators leading the market: ArgoCD and FluxCD. Both apply GitOps principles in different ways.

1. ArgoCD (Unified Architecture with a UI) #

ArgoCD is designed as a feature-rich GitOps platform equipped with a very informative graphical interface (Web UI) for visualizing Kubernetes resource dependency trees.

• Application Controller: The main agent running in the cluster to monitor the actual resource state and compare it with the target manifests in Git.
• Repo Server: The internal component tasked with cloning Git repositories, rendering manifests (Kustomize/Helm), and generating clean YAML manifests.
• Lifecycle Statuses:
  • Synced: The cluster’s actual state is identical to the Git repository state.
  • OutOfSync: A difference in property values or object counts is detected between Git and the cluster.
  • Healthy: All Kubernetes objects run normally (e.g. Pods in Running status, Services with Endpoints).
  • Degraded: Objects suffer operational failures (e.g. Pods stuck in CrashLoopBackOff).

2. FluxCD (Modular Architecture & GitOps Toolkit) #

FluxCD (often called Flux) takes a very different approach. Flux is designed with the UNIX philosophy: modular, no built-in graphical UI, lightweight, and works purely as Custom Resource Definition (CRD) controllers inside the cluster.

Flux divides its tasks into several dedicated controllers (the GitOps Toolkit):

• Source Controller: Specifically watches and pulls data from various external sources (Git repositories, Helm charts, S3 buckets).
• Kustomize Controller: Responsible for reconciling and applying Kustomize manifests to the cluster.
• Helm Controller: Specifically manages the Helm release lifecycle declaratively.

GitOps Workflow in Industry (GitOps Workflow Loop) #

Mature enterprise-level GitOps implementation requires us to separate code repositories into two types: the Application Code Repository and the Environment Manifest Repository (Environment Repo).

[!IMPORTANT] Why Must We Separate Repositories? If we combine application code and Kubernetes manifests in the same Git repository: when the CI pipeline finishes pushing a new image and updates the image tag in the manifest, it triggers a new commit in the repository. That new commit re-triggers the CI pipeline from the start, creating an infinite build loop. Repository separation completely prevents this problem.

sequenceDiagram
    participant Dev as Developer
    participant AppRepo as Application Git Repo
    participant CI as CI Pipeline (GitHub Actions)
    participant Registry as Container Registry
    participant ConfigRepo as Config Git Repo (GitOps)
    participant Argo as ArgoCD Controller
    
    Dev->>AppRepo: Push new application code (PR Merged)
    AppRepo->>CI: Trigger the build pipeline
    CI->>CI: Run unit tests & build the image
    CI->>Registry: Push the Docker image (tag: commit-sha)
    CI->>ConfigRepo: Update the image tag in overlays/production/kustomization.yaml
    Note over ConfigRepo: New commit detected in the Config Repo
    Argo->>ConfigRepo: Pull the latest manifests
    Argo->>Argo: Detect deviation (OutOfSync)
    Argo->>Argo: Apply 'selfHeal' & 'prune'
    Note over Argo: Cluster synced (Synced & Healthy)

ArgoCD Application Manifest (Production Readiness) #

Here’s a production-ready ArgoCD Application custom resource manifest example managing the payment-gateway application with self-healing and automatic deletion (pruning) policies:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-gateway-production
  namespace: argocd
  finalizers:
    # Guarantees all cluster resources are cleanly deleted if this Application object is removed
    - resources-finalizer.argocd.argoproj.com
spec:
  project: default

  # Configuration source (Git Repo)
  source:
    repoURL: 'https://github.com/company-org/k8s-config-manifests.git'
    targetRevision: main # Watches the main branch
    path: apps/payment-gateway/overlays/production # The target Kustomize path

  # Deployment destination (Target cluster)
  destination:
    server: 'https://kubernetes.default.svc' # The local cluster where ArgoCD runs
    namespace: payment

  # Synchronization Policy
  syncPolicy:
    automated:
      # Deletes cluster resources if their definitions are removed from Git
      prune: true
      # Auto-rewrites if manual modifications (kubectl edit) happen directly in the cluster
      selfHeal: true
    syncOptions:
      # Allows ArgoCD to create the namespace if it doesn't exist yet
      - CreateNamespace=true
      # Applies manifest schema validation before applying
      - Validate=true
      # Does sync optimization gradually (sync waves)
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5 # Retry up to 5 times if sync fails
      backoff:
        duration: 10s # Initial retry delay
        factor: 2 # The delay multiplier factor (exponential)
        maxDuration: 5m

Synchronization Policies: Automatic vs Manual Approval Gates #

Applying full automatic sync (auto-sync) across all working environments has different security trade-offs.

1. Development & Staging Environments #

• Strategy: Auto-Sync + Self-Healing Active.
• Philosophy: Iteration speed is the top priority. Developers can see the results of their code commits on the staging cluster within minutes without administrative hurdles.

2. Production Environment #

• Strategy: Manual Sync / Git Pull Request Gate.
• Philosophy: Security and stability come first. We can configure ArgoCD in production without syncPolicy.automated. When a new commit enters the main branch in the GitOps repository, ArgoCD shows OutOfSync status (colored yellow). Cluster operators must review the differences (visual diff) through the ArgoCD UI, coordinate with the team, and press the Sync button manually when ready (manual approval gate).

An alternative is still enabling auto-sync in production, but restricting releases through Git Branching / Git Tags mechanisms. The CI pipeline only writes manifests to the release-production branch after approval via a Git Pull Request, so Git acts as the approval gate.

Comprehensive Comparison: ArgoCD vs FluxCD #

The table below presents detailed architectural and operational difference mappings between the two leading GitOps tools:

*Note: FluxCD can be combined with third-party tools like Weave GitOps to get a UI, but it isn’t built-in.

Anti-Patterns vs Best Solutions #

Let’s study the most critical operational mistakes when applying GitOps in Kubernetes clusters along with their fixes.

Anti-Pattern 1: Storing Plaintext Credentials in GitOps Repositories #

Storing raw Secret YAML files or .env files containing production passwords in the Git repository under the justification that all repositories are in a private organization.

Consequences #

Those credentials get permanently exposed to Git history. Developers with GitOps repository access can see production access keys.

Best Solution #

Integrate the Git repository with in-cluster decryption operators like Mozilla SOPS or Bitnami Sealed Secrets. Manifests committed to Git are in a fully safe encrypted state. Only operators inside the production Kubernetes cluster hold the private keys to decrypt that data into regular Secret objects.

Anti-Pattern 2: Doing Manual Modifications (Direct Cluster Modification) #

Cluster operators run kubectl edit, kubectl apply, or kubectl delete commands directly on the production cluster to fix emergency problems (hotfixes), without writing those changes into the Git repository.

Consequences #

• If the selfHeal: true feature is active on ArgoCD: ArgoCD detects that difference as a deviation and overwrites our manual modification within seconds, deleting our emergency fix and returning the bug.
• If selfHeal is off: The cluster suffers configuration drift. When the next deployment is triggered through Git, our emergency change gets deleted without a trace, triggering the bug’s return later.

Best Solution #

All changes must go through the Git gate. In emergency conditions, make a fix commit in Git, merge it, and let the GitOps operator distribute it to the cluster. If we’re forced to take direct cluster action to save the system from total failure, make sure we immediately write those changes into the Git repository as soon as the system stabilizes.

Anti-Pattern 3: Using Helm Releases Without Pinning Locked Versions (Dynamic Helm Versioning) #

Pointing the GitOps repository to an external Helm chart using dynamic version tags like * or latest.

# ✗ ANTI-PATTERN: Using dynamic versions on Helm sources
source:
  chart: order-api
  repoURL: https://charts.company.com
  targetRevision: "*" # ← DANGEROUS: Randomly fetches the latest version on sync

Best Solution #

Always pin a specific, immutable version on the Helm chart target revision in our GitOps manifests to guarantee release consistency.

# ✓ SOLUTION: Declaratively pin the Helm chart version
source:
  chart: order-api
  repoURL: https://charts.company.com
  targetRevision: "1.4.2" # ← Safe, audited, and reproducible

GitOps Readiness Review Checklist #

Use the following audit checklist to make sure our GitOps delivery pipeline is ready to operate safely in production:

REPOSITORY & ACCESS SEGREGATION:
  □ The application code repository is physically separated from the cluster configuration manifest repository.
  □ Cluster admin write credentials (kubeconfig) have been removed from all external CI/CD runner servers.
  □ Commit/push access rights to the 'main' branch of the GitOps repository are protected by a minimum 2-person review rule (Pull Request protection).

IN-CLUSTER SECURITY HARDENING:
  □ All Secrets committed to GitOps repositories are encrypted using Mozilla SOPS or Sealed Secrets.
  □ RBAC policies for the ArgoCD/FluxCD service accounts are configured with least privilege permissions.
  □ The 'selfHeal' and 'prune' features are enabled in staging to test container self-healing resilience.

OPERATIONAL & DRIFT STRATEGY:
  □ Operator teams understand the rollback flow using the 'git revert' tactic (not kubectl rollout undo).
  □ Integration notifications (like sending ArgoCD sync status to a Slack channel) are active.
  □ Production sync policies use manual approval gate mechanisms or controlled git tagging.

Summary #

• Git as the only truth — The Kubernetes cluster’s actual state must always precisely reflect what’s written in the GitOps repository.
• Pull-based secures the cluster — Protect our production clusters by eliminating external kubeconfig access; let in-cluster operators (ArgoCD/FluxCD) pull manifests from Git.
• Separate code & configuration repositories — Prevent infinite build loops by separating the application source code repository from the GitOps manifest repository.
• Enable selfHeal and prune — Configure automatic sync policies so operators can dynamically delete stale objects and overwrite un-audited manual modifications.
• Do rollbacks via git revert — Learn the GitOps incident recovery flow: rollbacks are done by canceling manifest commits in Git, not running ad-hoc CLI commands in the cluster.
• Secure Secrets in repositories — Never put plaintext passwords in Git; use Sealed Secrets or Mozilla SOPS to store sensitive data encrypted in GitOps repositories.

← Previous: Rollback Strategy   Next: Deployment Anti-Patterns →