Introduction

Kubernetes, a powerful orchestration platform for containerized applications, deals with the dynamic nature of distributed systems where pods are frequently created and deleted. Monitoring recently deleted pods can be essential for diagnosing issues, ensuring stability, and auditing purposes. This article will explore how to list recently deleted pods in a Kubernetes cluster, leveraging Kubernetes API tools and techniques.

Understanding Pod Lifecycle in Kubernetes

Before diving into how to list recently deleted pods, it's important to comprehend the lifecycle of a Kubernetes pod, which includes several phases:

1. Pending: The pod has been accepted by the Kubernetes system, but one or more of the containers has not been created yet. This includes time spent waiting for the scheduler to determine on which node it should run.
2. Running: The pod has been bound to a node, and all containers have been created. At least one container is still running, or is in the process of starting or restarting.
3. Succeeded: All containers in the pod have terminated successfully, and there are no containers restarting.
4. Failed: All containers in the pod have terminated, and at least one container has terminated in failure.
5. Unknown: For some reason, the state of the pod could not be determined.

Pods also have conditions that define the “Ready” state, reflecting if a pod is fit to accept traffic. It's noteworthy that when pods are deleted, they undergo a termination phase before removal.

Listing Recently Deleted Pods

To specifically target recently deleted pods, Kubernetes does not directly offer a historical view of deleted pods through native commands like kubectl get pods. Instead, options include:

1. Leveraging Events

Kubernetes events can provide insights into the lifecycle changes for pods, including deletions:

kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'

This command lists all events sorted by timestamp. By filtering for specific event reasons such as Killing, you can check for recent pod deletions:

kubectl get events --all-namespaces --field-selector=reason=Killing --sort-by='.metadata.creationTimestamp'

2. Utilizing the Drains Node Feature

Pods often get deleted as part of a node drain operation, typically during maintenance windows or updates:

kubectl drain <node-name> --ignore-daemonsets --delete-local-data

The command cleans any pods scheduled on the specified node, and they will register as deleted pods during this action.

3. Using Logs and Audit Events

If audit logging is enabled, Kubernetes audit logs can provide detailed records of API requests and responses, including the deletion of pods:

• Access the logs stored based on your backend log management (e.g., Elasticsearch, Cloudwatch).
• Query for deleted resources by searching logs with relevant API calls like Delete on pods resources.

4. Third-party Tools

Several tools may also trace the lifecycle of pods and note deletion events:

• Prometheus with Grafana: By scraping Kubernetes metrics, you can create alerts and dashboards that visualize pod creation and termination patterns.
• Fluentd or ELK Stack: These logging pipelines can help analyze logs for delete operations.

Example: Monitoring with Prometheus

Configure Prometheus to monitor pod lifecycle events. This requires setting up metrics such as:

• kube_pod_status_phase
• kube_pod_deletion_timestamp

These metrics can be queried in Prometheus, allowing for constructing alerts or graphs indicating recent pod deletions.

1- alert: PodDeletionAlert
2  expr: kube_pod_deletion_timestamp{namespace="<target_namespace>"} > time() - 3600
3  for: 5m
4  labels:
5    severity: "warning"
6  annotations:
7    summary: "Pod recently deleted in the last hour"
8    description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has been deleted recently."

Summary Table

Conclusion

Monitoring recently deleted pods involves understanding Kubernetes events, utilizing system logs, and leveraging third-party tools to gain insights into the ephemeral nature of pods. By implementing these strategies, teams can maintain robust oversight and perform proactive maintenance and debugging within their Kubernetes environments.