Overview

Kubernetes Jobs are a crucial component for running batch processing tasks that require completion. They are typically initiated through a CronJob, which is a specialized job that runs on a time-based schedule. However, managing and cleaning up completed Jobs can become a significant responsibility, particularly in environments with numerous CronJobs. Efficiently removing completed Jobs can free system resources and simplify the operational environment.

This article explores various strategies for removing completed Kubernetes Jobs automatically. We'll delve into technical configurations and practical scripts to facilitate this cleanup process.

Understanding Kubernetes Job and CronJob

To begin, you must understand the distinction between a Job and a CronJob:

• Job: A Kubernetes Job resource creates one or more Pods and ensures a specified number of them terminate successfully.
• CronJob: A CronJob schedules a Job to run periodically based on the defined schedule using Linux cron syntax.

The Challenge

The aforementioned components are well-optimized for scheduling and executing tasks, but by default, Kubernetes retains completed Jobs until they are manually deleted. As these Jobs accumulate over time, they can collectively become redundant and clog up the system.

Solutions for Automatic Cleanup

1. Use the ttlSecondsAfterFinished field
   Starting in Kubernetes 1.12, the ttlSecondsAfterFinished field is available for Jobs. It defines the time-to-live (TTL) for a Job after it has finished successfully. You can specify the TTL in seconds, and once it is exceeded, the Job will automatically be deleted.

1   apiVersion: batch/v1
2   kind: Job
3   metadata:
4     name: example-job
5   spec:
6     ttlSecondsAfterFinished: 3600 # 1 hour
7     template:
8       spec:
9         containers:
10         - name: example
11           image: busybox
12           command: ["sleep", "10"]
13         restartPolicy: Never

1. CronJob Garbage Collection with successfulJobsHistoryLimit and failedJobsHistoryLimit
   CronJobs offer built-in configuration options to determine how many completed and failed Jobs should be retained. These settings help manage the number of Jobs without writing additional scripts.

1   apiVersion: batch/v1beta1
2   kind: CronJob
3   metadata:
4     name: example-cronjob
5   spec:
6     schedule: "*/5 * * * *"
7     jobTemplate:
8       spec:
9         template:
10           spec:
11             containers:
12             - name: example
13               image: busybox
14               command: ["sleep", "10"]
15             restartPolicy: Never
16     successfulJobsHistoryLimit: 3
17     failedJobsHistoryLimit: 1

1. Custom Cleanup Scripts
   For more customized cleanup strategies, Kubernetes administrators can employ scripts executed periodically as Jobs. These scripts can use the Kubernetes API to list and delete completed Jobs based on specific criteria.

1   # Custom script example to delete completed jobs
2   kubectl get jobs --namespace default --field-selector status.successful=1 --output=json | \
3   jq '.items[] | select(.status.completionTime | fromdateiso8601 < (now - 3600)) | .metadata.name' | \
4   xargs -I {} kubectl delete job {}

Example Scenario

Imagine an environment where nightly batch processing is scheduled using CronJobs. Each Job completes successfully within the scheduled interval, leading to the accumulation of completed Jobs. Using ttlSecondsAfterFinished ensures that any residual data and resources are cleared an hour after completion, while successfulJobsHistoryLimit keeps the most recent three successful Jobs, ensuring administrators have enough historical Logs without overwhelming the system.

Summary Table

Conclusion

Managing completed Jobs created by CronJobs in Kubernetes is essential for maintaining a clean and efficient cluster environment. The built-in features such as ttlSecondsAfterFinished and history limits provide automatic cleanup with minimal configuration. For more complex requirements, custom scripts can be implemented using Kubernetes Jobs themselves. Each approach provides different levels of ease, automation, and control, catering to varying operational needs. By leveraging these solutions, Kubernetes administrators can significantly reduce clutter and ensure resources remain available for active Jobs and other components.