Canary Deployment #

Launching new features to production always carries uncertainty. Even though we’ve done thorough testing in the Staging environment, system behavior under heterogeneous, massive real user traffic often triggers unexpected anomalies. If we use the Blue/Green strategy, we immediately expose 100% of users to the new version after the cutover button is pressed. If a critical bug slips past initial testing, all active users feel its impact before we can roll back.

The Canary Deployment strategy is designed to solve this problem by applying the progressive delivery principle. Inspired by old coal miner practices of carrying canaries into mine tunnels to early-detect toxic gas, we first release the new application version (v2) to a small subset of container Pods. We then redirect a small portion of real user traffic (e.g. 2% or 5%) to those Pods while intensively monitoring application performance metrics. If the Canary version proves safe and stable, we gradually increase the traffic portion until reaching 100%.

The Canary Implementation Spectrum in Kubernetes #

We can implement Canary Deployment in Kubernetes through several approach levels, from simple manual methods based on Pod replica count manipulation to advanced automation using dedicated controllers.

+-----------------------------------------------------------------------------+
| CANARY IMPLEMENTATION LEVELS:                                              |
|                                                                             |
| 1. Simple (L3/L4 Service Pod Ratio)                                         |
|    - Coarse traffic division based on the Pod count ratio.                  |
|    - Example: 9 v1 Pods + 1 v2 Pod = ~10% Traffic.                          |
|                                                                             |
| 2. Medium (L7 Ingress Controller Weighting)                                 |
|    - Precise traffic division at the HTTP Proxy (Ingress) level.            |
|    - Example: NGINX Ingress annotation canary-weight: "5".                  |
|                                                                             |
| 3. Advanced (Automated Progressive Delivery - Argo Rollouts)                |
|    - Full automation based on Prometheus metrics + auto-rollback.           |
|    - Evaluates p99 latency & HTTP error rate per minute.                    |
+-----------------------------------------------------------------------------+

Approach 1: Traffic Division Based on Pod Replica Ratios (Layer 4) #

This approach is the most basic method, requiring no tools outside built-in Kubernetes. We create two separate Deployment objects sharing the same identity labels, so a global Service distributes traffic round-robin to all Pods from both versions.

1. Stable Deployment Manifest (v1) — Controlling 90% of Traffic #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service-stable
  namespace: production
spec:
  replicas: 9 # ← 9 active Pods
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
        track: stable
    spec:
      containers:
      - name: api
        image: company/api-service:v1.0.0
        ports:
        - containerPort: 8080

2. Canary Deployment Manifest (v2) — Controlling 10% of Traffic #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service-canary
  namespace: production
spec:
  replicas: 1 # ← 1 active Pod
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
        track: canary
    spec:
      containers:
      - name: api
        image: company/api-service:v1.1.0 # New Version
        ports:
        - containerPort: 8080

3. Traffic-Unifying Service Manifest #

apiVersion: v1
kind: Service
metadata:
  name: api-service-global
  namespace: production
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: api-service # ← Matches the 'app: api-service' label from both Deployments

Key Limitations of This Approach: #

• Coarse Granularity: If we want to divert only 1% of traffic to Canary, we’re forced to run at least 99 Stable Pods and 1 Canary Pod. This triggers massive infrastructure resource waste.
• Distribution Inaccuracy: Because Kubernetes Service routing works with random round-robin (kube-proxy), the traffic percentage flowing to Canary containers isn’t guaranteed to be exactly 10%.

Approach 2: Precise Routing Using NGINX Ingress Weighting (Layer 7) #

To overcome the Pod division method’s limitations above, we can leverage the Layer 7 routing capability of the NGINX Ingress Controller. With this approach, we create two standalone Service objects (one for Stable, one for Canary) and define an additional Ingress with weight control annotations (canary-weight).

flowchart LR
    Traffic["User Traffic (api.company.com)"] --> Ingress["Ingress Controller"]
    Ingress -->|"95% Traffic"| ServiceStable["Stable Service"] --> PodV1["v1 Pods"]
    Ingress -->|"5% Traffic"| ServiceCanary["Canary Service"] --> PodV2["v2 Pods"]

1. Main Ingress Manifest (Stable) #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress-stable
  namespace: production
spec:
  ingressClassName: nginx
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service-stable
            port:
              number: 80

2. Canary-Specific Ingress Manifest (Managing Traffic Division) #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress-canary
  namespace: production
  annotations:
    # Instructs NGINX to treat this Ingress as a Canary route
    nginx.ingress.kubernetes.io/canary: "true"
    # Diverts exactly 5% of HTTP traffic to the Canary Service
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  ingressClassName: nginx
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service-canary
            port:
              number: 80

We can dynamically update the nginx.ingress.kubernetes.io/canary-weight annotation value (e.g. raising it to "10", "25", or "50") through CI/CD pipeline scripts without changing the application Pod replica capacity.

Approach 3: Progressive Delivery Automation with Argo Rollouts #

In industry-scale production environments, manually managing traffic percentage transitions and monitoring metrics is an error-prone action. Argo Rollouts is a Kubernetes Custom Controller replacing the built-in Deployment object with a new object called Rollout.

Argo Rollouts automates the entire Canary cycle: gradually increasing traffic, watching the Prometheus server to check query success rates (HTTP success rate), and automatically triggering instant release cancellation (rollback) if Canary performance degrades.

flowchart TD
    Start["New Deployment via GitOps (v2)"] --> Step1["Argo Rollouts sets the weight to 5%"]
    Step1 --> Step2["Wait 10 minutes (Metric Observation)"]
    Step2 --> Dec1{"Does the Prometheus query<br>show an error rate < 1%?"}
    
    Dec1 -- "No (Anomaly)" --> RollbackAuto["INSTANT AUTOMATIC ROLLBACK<br/>'(Set weight to 0%, remove v2)'"]
    Dec1 -- "Yes (Normal)" --> Step3["Scale Up the weight to 25%"]
    
    Step3 --> Step4["Wait 10 minutes (Metric Observation)"]
    Step4 --> Dec2{"Does the Prometheus query<br>show an error rate < 1%?"}
    
    Dec2 -- "No" --> RollbackAuto
    Dec2 -- "Yes" --> Step5["Scale Up the weight to 100% (Release Complete)"]
    
    style RollbackAuto stroke:#d32f2f,stroke-width:2px
    style Step5 stroke:#388e3c,stroke-width:2px

1. Rollout Manifest (Replacing the Deployment) #

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-api
  namespace: production
spec:
  replicas: 10
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      containers:
      - name: order-api
        image: company/order-api:v2.0.0 # New Canary candidate version
        ports:
        - containerPort: 8080
  strategy:
    canary:
      canaryService: order-api-canary-svc # The Canary reference Service
      stableService: order-api-stable-svc # The Stable reference Service
      trafficRouting:
        nginx:
          stableIngress: order-api-ingress # The main Ingress to be manipulated
      steps:
      - setWeight: 5
        # Wait 5 minutes for initial metric observation
      - pause: {duration: 5m}
      - analysis:
          # Runs automatic metric evaluation based on the AnalysisTemplate
          templates:
          - templateName: http-error-analysis
      - setWeight: 20
      - pause: {duration: 10m}
      - analysis:
          templates:
          - templateName: http-error-analysis
      - setWeight: 50
      - pause: {} # Pause without duration = Wait for manual approval (manual approval gate)
      - setWeight: 100

2. AnalysisTemplate Manifest (Release Success Criteria) #

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-error-analysis
  namespace: production
spec:
  metrics:
  - name: success-rate
    interval: 1m # Check the metric every 1 minute
    successCondition: result[0] >= 0.995 # The HTTP success rate must be >= 99.5%
    failureLimit: 3 # Maximum tolerance of 3 consecutive failures before rollback
    provider:
      prometheus:
        address: http://prometheus-k8s.monitoring:9090
        query: |
          sum(rate(http_requests_total{app="order-api", status!~"5.."}[2m])) 
          / 
          sum(rate(http_requests_total{app="order-api"}[2m]))          

Critical Metrics for Canary Release Monitoring #

Running a Canary without adequate observability metrics is a dangerous action. We must identify the key metrics (Golden Signals) that truly distinguish Canary version failures from ordinary cluster traffic noise.

1. HTTP Error Rate (5xx Response Percentage) #

This metric is the most crucial for immediately catching internal server logic failures or database integration damage. Compare the HTTP 5xx status percentage on Canary containers against Stable containers.

• PromQL Query: sum(rate(nginx_ingress_controller_requests{status=~"5.."}[5m]))

2. p95 / p99 Latency (Latency Tail) #

Don’t use the average latency metric because performance degradation experienced by a small user subset is often masked by the global average value. Use the 95th or 99th percentile to detect extreme latency.

• PromQL Query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

3. Resource Saturation (CPU/Memory Saturation) #

New versions may have slow memory leak bugs not detected during initial initialization. Observe RAM consumption increases and the frequency of containers being killed for running out of memory (OOMKilled events).

The Importance of Statistical Significance and Traffic Volume #

What’s the ideal observation duration for each Canary phase? The answer heavily depends on our application’s transaction traffic volume (throughput).

• High Traffic (e.g. > 1000 requests per second): We get statistically sufficient data sample sizes within just 5 to 10 minutes of monitoring.
• Low Traffic (e.g. < 5 requests per second): Metric data doesn’t have adequate statistical significance in 10 minutes. Bugs may go undetected because no user triggered the problematic endpoint during the observation period. In this scenario, we must extend the Canary duration (e.g. up to 12 or 24 hours) or use traffic diversion techniques based on specific routes (header routing) to test functions in a targeted way.

Anti-Patterns vs Best Solutions #

Let’s study common mistakes when designing Canary Deployments in Kubernetes clusters along with their best fixes.

Anti-Pattern 1: Canary Without Sticky Sessions (Traffic Flapping) #

Launching a Canary on an e-commerce web application without configuring session persistence (session affinity) at the Ingress level. Users in the middle of checkout get randomly diverted between v1 (Stable) and v2 (Canary) Pods on every page click. If v2 has a different session schema structure, users suddenly get logged out or experience shopping cart failures (cart loss).

Best Solution #

Use the Canary Affinity feature on the Ingress Controller. This forces the Ingress to send a special cookie to the user’s browser after they’re first diverted to the Canary route, guaranteeing that the user’s subsequent transactions always land on the same Canary version.

# ✓ SOLUTION: Using a Canary Cookie for session persistence
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress-canary
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
    # Forces cookie-based session persistence
    nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
    nginx.ingress.kubernetes.io/canary-by-cookie: "always"
spec:
  # ... spec Backend Service ...

Anti-Pattern 2: Ignoring Downstream Dependency Testing (Database Leak Blast Radius) #

Running a Canary for microservice A which dynamically writes new data to a shared database. The v2 code turns out to write data in a wrong binary format. Although microservice A’s Canary version appears to run normally without HTTP 5xx errors, our database has already been polluted with corrupted data. This causes microservice B (Downstream Service) to suffer mass crashes on the other side.

Best Solution #

Enable holistic system-level monitoring. Analyze database query metrics (like locked rows, active transactions) and observe downstream service error logs during the Canary promotion process. Disciplined use of backward-compatible data architecture schemas is essential.

Complete Production Canary Manifests (NGINX Ingress Annotations) #

Here’s a complete production manifest example applying Canary deployment based on the NGINX Ingress Controller to divert exactly 10% of user traffic to the new version:

# 1. Service for the Stable Application
apiVersion: v1
kind: Service
metadata:
  name: catalog-service-stable
  namespace: product-catalog
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: catalog-app
    version: stable
---
# 2. Service for the Canary Application (New Version)
apiVersion: v1
kind: Service
metadata:
  name: catalog-service-canary
  namespace: product-catalog
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: catalog-app
    version: canary
---
# 3. Main Ingress (Routing 90% of Normal Traffic)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: catalog-ingress-stable
  namespace: product-catalog
spec:
  ingressClassName: nginx
  rules:
  - host: catalog.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: catalog-service-stable
            port:
              number: 80
---
# 4. Canary Ingress (Routing 10% of Dedicated Traffic to the Canary Service)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: catalog-ingress-canary
  namespace: product-catalog
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    # Divides traffic at exactly 10% precisely at the NGINX Proxy level
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: catalog.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: catalog-service-canary
            port:
              number: 80

Canary Readiness Review Checklist #

Use the following audit checklist to make sure our Canary release pipeline is ready before launching a release in production:

SESSION & CONNECTION PERSISTENCE:
  □ Session persistence cookies or headers (session affinity) are configured to prevent traffic flapping.
  □ The Canary application uses isolated namespaces and Services for easier tracking.
  □ Downstream services are confirmed unaffected by the new payload types from the Canary version.

MONITORING & PROMETHEUS METRICS:
  □ PromQL queries for calculating HTTP error rates and p99 latency are validated for accuracy.
  □ Anomaly detection thresholds are determined (e.g. maximum 0.5% error rate during releases).
  □ Metric query intervals are balanced (not too frequent to avoid overloading the Prometheus server).

ROLLBACK & PROMOTE PROCEDURES:
  □ CI/CD pipeline scripts are ready to gradually raise the weight percentage (canary-weight).
  □ Promotion failure timeout limits (failure limits) are set in Argo Rollouts.
  □ The Canary Ingress cleanup procedure at the end of a successful release is automatically configured.

Summary #

• Minimize release blast radii — Canary deployments protect production clusters by limiting new version bug exposure to only a small initial user subset (e.g. 2-5% of traffic).
• Use Ingress weighting for accuracy — Avoid manual traffic division using Pod replica ratios; leverage the canary-weight annotation on Ingress Controllers for precise traffic percentage division.
• Automate with Argo Rollouts — Use the Rollout CRD for fully automated progressive delivery that watches Prometheus metrics and does auto-rollback when anomalies occur.
• Watch p99 percentile metrics — Don’t be fooled by average latency values; observe p99 latency to catch extreme delays on user requests.
• Apply Canary Session Affinity — Configure cookie-based routing to maintain active user session consistency so they aren’t randomly diverted between Pods.
• Adjust durations by throughput — Low-traffic applications need longer Canary observation periods to obtain valid statistical significance.

← Previous: Blue/Green Deployment   Next: Recreate →