Job & CronJob #

In a Kubernetes cluster, not all workloads are designed to stay running endlessly. If web API applications, gateways, and databases must stand ready serving traffic every second, there’s another kind of task that’s transient — running to process something, completing successfully, then cleanly dying. Examples include running database migrations, generating monthly financial reports, sending mass promotional emails, or processing video file queues. To handle these run-to-completion tasks, Kubernetes provides the Job and CronJob objects.

Using a Deployment for transient tasks is a fatal mistake because a Deployment keeps trying to revive containers that already completed successfully. Understanding the internal architecture of Jobs and CronJobs, parallel processing parameters, and failure handling policies is crucial for automating large-scale batch processing efficiently in production clusters.

Batch Workloads: The Run-to-Completion Philosophy #

The fundamental difference between long-running services and batch processing lies in how the container exit code is interpreted:

  • Deployment: Treats an application container exiting (whether with exit code 0 or not) as an anomaly. The Deployment Controller always orders the Kubelet to restart the container to maintain service availability.
  • Job: Has a different understanding. If the main container in the Pod exits with code exit 0 (success), the Job Controller terminates that Pod’s entire lifecycle, marks it as Completed, and doesn’t try to restart it.

Here’s a visualization of the Job status transition flow from creation to automatic cleanup by the Kubelet:

flowchart TD
    Created["Job Created"] --> Schedule["Pod Scheduled"]
    Schedule --> Running["Pod Running"]
    
    Running --> ExitCheck{"Did the Container\nExit with Code 0?"}
    ExitCheck -- "Yes" --> Succeeded["Job Completed Successfully (Succeeded)"]
    ExitCheck -- "No" --> RestartPolicyCheck{"restartPolicy?"}
    
    RestartPolicyCheck -- "OnFailure" --> RestartContainer["Restart the Container in the Same Pod"]
    RestartContainer --> Running
    
    RestartPolicyCheck -- "Never" --> NewPod["Create a New Pod on a Different Node"]
    NewPod --> BackoffCheck{"backoffLimit\nReached?"}
    
    BackoffCheck -- "Not yet" --> Running
    BackoffCheck -- "Yes" --> Failed["Job Declared Failed (Failed)"]
    
    Succeeded --> TTL{"ttlSecondsAfterFinished\nReached?"}
    Failed --> TTL
    
    TTL --> Delete["Automatically Delete the Job & Pods from etcd"]

The Job Object: Handling Run-to-Completion Tasks #

The Job object ensures one or more Pods complete their tasks thoroughly.

Here’s a production-grade Job manifest example for running a safe database migration:

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migrator
  namespace: production
spec:
  completions: 1                  # The number of successful completions required
  parallelism: 1                  # The number of Pods allowed to run simultaneously
  backoffLimit: 4                 # Maximum failure tolerance before the Job gives up
  activeDeadlineSeconds: 600      # Maximum Job execution time limit (10 minutes)
  template:
    spec:
      restartPolicy: OnFailure    # OnFailure | Never (Always is STRICTLY FORBIDDEN)
      containers:
      - name: migrator-app
        image: company-app:v3.2.0
        command: ["/app/bin/migrate", "up"]
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "1Gi"

Choosing restartPolicy: Never vs OnFailure #

In a Job manifest, writing restartPolicy: Always is strictly forbidden. We must choose one of two policy options:

  1. OnFailure: If a container inside the Pod exits with an error (non-zero exit code), the Kubelet restarts that container in the same Worker Node and Pod. This pattern is very efficient because it avoids new Pod rescheduling overhead.
  2. Never: If a container fails, the Kubelet doesn’t restart it. Instead, the Job Controller asks the Scheduler to create a new Pod on another Worker Node to continue the failed task. This pattern is ideal when the failure was caused by physical damage to the host node system.

Advanced Failure Handling Policy (podFailurePolicy) #

In modern Kubernetes versions (v1.25+), we can use the podFailurePolicy feature to handle execution failures far more precisely than relying on a global backoffLimit. Without this policy, every Pod failure — whatever the cause (application code bug, OOM, or infrastructure failure) — counts the same and consumes the backoffLimit allowance.

With podFailurePolicy, we can differentiate actions based on the container’s Exit Code or Pod Failure Conditions:

spec:
  backoffLimit: 6
  podFailurePolicy:
    rules:
    - action: FailJob             # Fail the Job IMMEDIATELY if the exit code is 42 (e.g. Fatal Error)
      onExitCodes:
        containerName: migrator-app
        operator: In
        values: [42]
    - action: Ignore              # Don't count this failure toward backoffLimit if the Pod was evicted (e.g. node maintenance)
      onPodConditions:
      - type: DisruptionTarget    # Indicates the Pod was stopped due to external disruption

Benefits of Implementing podFailurePolicy: #

  • Saves Resources & Time: If the application detects a configuration error or database credential error (producing a specific exit code, e.g. 42), there’s no point in Kubernetes retrying the Job 6 more times. The Job gets marked failed instantly, speeding up the CI/CD feedback loop.
  • Resilience to Infrastructure Disruption: If a Pod dies because the node underwent spot instance eviction or maintenance scaling, we don’t want that counted as an application error. With the Ignore rule, the Job retries without consuming our failed attempt quota.

Design Patterns Combining Completions and Parallelism #

We can design various batch data processing execution patterns by combining the completions (success target) and parallelism (concurrency capacity) property values:

  • Single Job: completions: 1 and parallelism: 1. Runs one single Pod to completion. Perfect for DB schema initialization.
  • Sequential Batch: completions: 5 and parallelism: 1. Runs 5 tasks one after another. The 2nd Pod is only created after the 1st completes successfully.
  • Large-Scale Parallel Batch: completions: 100 and parallelism: 10. Runs 10 Pods concurrently until reaching 100 total successful executions.
  • Work Queue: completions unset (left empty) and parallelism: 5. Runs 5 Pods concurrently. Each Pod dynamically pulls data from an external queue server (like RabbitMQ). The Job is declared complete once the queue is empty and all containers exit successfully.

Using the Modern Scheduling Index (JOB_COMPLETION_INDEX) #

Since Kubernetes 1.21+, if we use the Indexed Job type (completionMode: Indexed), Kubernetes automatically injects a unique index number into each Pod under the batch.kubernetes.io/job-completion-index annotation (from 0 to completions-1).

Our application can read this index through the Downward API to divide the data processing range evenly:

# Pod index 0 processes data IDs 1-1000, Pod index 1 processes IDs 1001-2000, etc.
env:
- name: MY_WORK_INDEX
  valueFrom:
    fieldRef:
      fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

Automatic Cleanup Policy (TTL Controller) #

By default, after a Job finishes successfully (Succeeded) or fails (Failed), the Job object and its descendant Pods stay in the cluster’s etcd database with Completed or Error status. This is intentional so administrators can read output logs or describe the failure.

However, in busy CI/CD environments running hundreds of Jobs daily, letting thousands of Completed Pods pile up clutters the CLI view and burdens Kubelet cache performance on host nodes.

To automate cleanup, we must use the ttlSecondsAfterFinished property:

spec:
  ttlSecondsAfterFinished: 3600 # Automatically delete the Job and all Completed Pods 1 hour after finishing

The CronJob Object: Automating Scheduled Tasks #

CronJob wraps the Job object so it can run periodically using standard Linux Cron schedule syntax.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-db-cleanup
  namespace: production
spec:
  schedule: "0 2 * * *"                 # Runs every day at 02:00 AM
  timeZone: "Asia/Jakarta"              # Sets the cluster's local timezone (K8s 1.25+)
  concurrencyPolicy: Forbid             # Controls execution overlap rules
  startingDeadlineSeconds: 180          # Late start tolerance limit (3 minutes)
  successfulJobsHistoryLimit: 3         # Keep logs of the last 3 successful Jobs
  failedJobsHistoryLimit: 1             # Keep logs of the last 1 failed Job
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: clean-agent
            image: db-utils:v1.0
            command: ["/app/clean-expired-sessions"]

Concurrency Policy on CronJobs #

One critical problem with scheduled tasks is when a Job execution runs so slowly that the next scheduled time arrives before the old Job finishes. We must control this behavior with concurrencyPolicy:

Three Concurrency Policy Options: #

  1. Allow (Default): Allows several Jobs to overlap and run simultaneously. This is dangerous if the task modifies the same database concurrently.
  2. Forbid: If the previous Job hasn’t finished, the new schedule is skipped. This is the safest choice for cleanup or database backup tasks.
  3. Replace: If the old Job is still running, Kubernetes immediately kills (SIGKILL) the old Job and replaces it with the newly triggered Job.

Here’s a visualization of the behavioral difference between the Forbid and Replace policies:

flowchart TD
    subgraph ForbidPolicy["ConcurrencyPolicy: Forbid"]
        F_Start["Schedule 2 Time Arrives"] --> F_Check{"Is Job 1\nstill running?"}
        F_Check -- "Yes" --> F_Skip["Skip / Cancel Schedule 2\n(Let Job 1 finish)"]
        F_Check -- "No" --> F_Run["Run Job 2"]
    end

    subgraph ReplacePolicy["ConcurrencyPolicy: Replace"]
        R_Start["Schedule 2 Time Arrives"] --> R_Check{"Is Job 1\nstill running?"}
        R_Check -- "Yes" --> R_Kill["Send SIGKILL to kill Job 1"]
        R_Kill --> R_Run["Run Job 2"]
        R_Check -- "No" --> R_Run
    end

    style F_Skip stroke:#e74c3c,stroke-width:2px
    style R_Kill stroke:#e74c3c,stroke-width:2px

Late Start Tolerance Policy (startingDeadlineSeconds) #

If the cluster master node loses power exactly when a CronJob execution time arrives, that CronJob gets missed. After the cluster comes back up, the CronJob Controller counts how many schedules were missed.

  • startingDeadlineSeconds: Sets the tolerance time limit (in seconds) for running a missed Job. If we set startingDeadlineSeconds: 180, and the cluster recovers 2 minutes after the missed schedule, the Job still runs. But if the cluster only recovers 10 minutes later, that schedule is permanently skipped for data safety.

The 100 Missed Schedules Detection Limit (CronJob Suspension) #

There’s one internal Kubernetes CronJob Controller aspect production administrators must understand: the 100 Missed Schedules Limit Rule.

If we define a CronJob without setting startingDeadlineSeconds (or leave it empty), and for whatever reason (e.g. the cluster went totally down, or the controller-manager died) the CronJob Controller misses more than 100 consecutive schedules, then:

  1. The CronJob Controller stops scheduling that CronJob forever.
  2. The CronJob status is passively suspended, and we see an error message in the controller logs like: Cannot determine if job needs to be started....

Why Is This Limit Enforced? #

It’s an etcd memory protection. Without the 100 limit, if a CronJob is set to run every minute (* * * * *) and the cluster dies for 5 days, when it comes back the controller would try to instantly create more than 7,000 Jobs to pay off the “debt” of missed schedules. That would immediately crash the cluster from memory exhaustion.

The Practical Solution #

To avoid this permanent suspension on CronJobs running very frequently (e.g. every minute or every 5 minutes), we must always configure startingDeadlineSeconds. By setting this value (e.g. startingDeadlineSeconds: 200), Kubernetes only checks whether there are missed schedules within the last 200 seconds. If there are, it runs exactly one Job. This prevents cumulative calculations from breaking the 100 missed schedules threshold.

Anti-Patterns in Job & CronJob Management #

Fatal mistakes that often paralyze production clusters when managing batch tasks:

Anti-Pattern 1: Ignoring activeDeadlineSeconds (Jobs Hanging Forever) #

Deploying a Job without limiting the maximum execution duration.

ANTI-PATTERN: Running the database-sync Job Without activeDeadlineSeconds
// WHAT WE DO:
- Deploy a Job to sync data to an external API.
- The external API has a network outage, making the process in our container
  stuck waiting for a response (read timeout) indefinitely.

// THE CONSEQUENCES IN PRODUCTION:
- The Job Pod stays running and eats cluster memory allocation forever.
- Because the Job never finishes, the next CI/CD deployment pipeline hangs waiting for confirmation.
- We waste cloud VM compute costs on a container that's actually doing nothing.

✓ THE RIGHT SOLUTION:
- Always declare a realistic time limit in the `activeDeadlineSeconds` property at the Job spec level:
  spec:
    activeDeadlineSeconds: 300 # If the Job isn't done in 5 minutes, force-stop all container processes
- Write timeout handling inside your main application code.

Anti-Pattern 2: Running CronJobs Too Frequently Without Limiting History #

Using a Kubernetes CronJob as a micro-scheduler (every few seconds) while storing unlimited history.

ANTI-PATTERN: Writing schedule: "* * * * *" Without successfulJobsHistoryLimit
// WHAT WE DO:
- Run a cleanup CronJob every 1 minute.
- Skip filling the `successfulJobsHistoryLimit` property (using the default of 3).

// THE CONSEQUENCES IN PRODUCTION:
- In 1 day, the cluster creates 1440 Job objects and 1440 Completed Pods.
- The cluster's etcd database bloats drastically, degrading overall Kubernetes API Server response performance.
- The `kubectl get pods` CLI floods with thousands of useless Completed Pod logs.

✓ THE RIGHT SOLUTION:
- Always strictly limit successful and failed Job history storage in the CronJob manifest:
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
- If the task must run within seconds, don't use a Kubernetes CronJob.
  Use a long-running application (Deployment) managing an internal queue (like Celery or BullMQ).

Summary #

  • The Run-to-Completion Philosophy — Jobs are specifically for transient tasks that finish once (run-to-completion), where an exit 0 code is interpreted as success.
  • Mandatory restartPolicy Rules — Write Never or OnFailure for the Job template spec restartPolicy; using the Always type is strictly forbidden.
  • Controlled Parallelism — Combine completions (success target) and parallelism (concurrent containers) values to design parallel batch processing workflows.
  • Efficient Data Indexing — Leverage the JOB_COMPLETION_INDEX on Indexed Job types to automatically divide work ranges among workers.
  • Automated TTL Cleanup — Configure ttlSecondsAfterFinished so the cluster automatically deletes Job objects and Completed Pods cluttering memory.
  • Forbid Concurrency Protection — Always use concurrencyPolicy: Forbid on CronJobs if the periodic task modifies the same database to avoid data corruption.
  • Secure the Maximum Duration — Apply activeDeadlineSeconds on every production Job to prevent unlimited container hangs (infinite timeout).

← Previous: DaemonSet   Next: Scheduler Workflow →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact