Introduction

Airflow scheduler logs can grow rapidly, so cleanup should be handled by retention policy and rotation automation rather than ad hoc deletion during incidents. In practice, the fastest path is to reduce the problem to a small reproducible baseline first, then reintroduce production constraints one by one. That approach keeps debugging local, prevents overfitting to one failing symptom, and makes your final implementation easier to explain to teammates.

Blindly removing logs can erase forensic data and break troubleshooting. The safer model is controlled retention by age, compressed archives, and separate policy for scheduler, worker, and task logs. A strong implementation separates configuration from execution flow, adds measurable checkpoints, and captures enough telemetry to distinguish transient failures from deterministic misconfiguration.

Core Sections

1) Define a narrow baseline before optimization

Start by identifying the smallest end-to-end version that should work reliably. Keep external dependencies minimal, remove optional features, and make defaults explicit. Once the baseline is stable, layer complexity gradually and verify behavior after each change. This staged workflow is more predictable than changing multiple variables at once and trying to infer root cause afterward.

2) Implement scheduled cleanup with age-based retention

1AIRFLOW_HOME=${AIRFLOW_HOME:-~/airflow}
2SCHED_LOG_DIR="$AIRFLOW_HOME/logs/scheduler"
3
4# Keep 14 days of scheduler logs
5find "$SCHED_LOG_DIR" -type f -name "*.log" -mtime +14 -print -delete
6
7# Optional: remove empty directories after cleanup
8find "$SCHED_LOG_DIR" -type d -empty -delete

This baseline snippet is intentionally conservative. It prioritizes readability, deterministic behavior, and explicit control points over clever shortcuts. For production, you can tune performance later, but first ensure the pipeline is correct and repeatable. If this step does not behave as expected, freeze further refactors and diagnose here; debugging gets exponentially harder once additional abstractions are layered on top.

3) Set rotation/cleanup behavior in deployment configuration

1[logging]
2base_log_folder = /opt/airflow/logs
3remote_logging = False
4
5[scheduler]
6child_process_log_directory = /opt/airflow/logs/scheduler
7
8# Pair with system logrotate or cron cleanup job to enforce retention.

Operational guardrails are what turn a working demo into a maintainable system. Add logging around key transitions, monitor latency and error classes, and define clear retry or fallback policy where failures are expected. Avoid silent recovery paths that hide data quality or state issues. Instead, emit structured signals that make post-incident analysis straightforward.

4) Validate behavior with repeatable checks

Run cleanup in dry-run mode first (-print without -delete) and confirm expected files. Then schedule execution during low-traffic windows and monitor disk usage trends weekly. Write a short verification checklist that can run in local development, CI, and pre-release environments. Include both success-path assertions and at least one intentional failure case. Over time, this checklist becomes regression protection: it documents assumptions, catches environment drift, and prevents future edits from reintroducing the same class of bug.

For teams maintaining this in production, add a short runbook that documents normal metrics, alert thresholds, and first-response steps. Operational clarity reduces mean time to recovery and lowers the cost of onboarding new contributors who need to troubleshoot the workflow quickly.

Common Pitfalls

• Deleting all logs immediately without an agreed retention policy.
• Applying one retention rule to scheduler logs and task logs with different operational needs.
• Running cleanup as a user without permission consistency, leaving partial deletions.
• Ignoring remote logging settings when logs are offloaded to object storage.
• Skipping dry-run checks before first production cleanup execution.

Summary

Airflow log hygiene should be policy-driven and automated so storage stays predictable without losing critical diagnostic history. The key pattern is consistent across stacks: keep the core path simple, instrument the edges, and validate with deterministic tests before scaling complexity.