Kubernetes provides robust tools for managing containerized applications across a cluster of machines. One often-encountered requirement in such systems is the ability to view the logs of terminated pods for troubleshooting or auditing purposes. In this article, we delve into various approaches and methods for achieving this.

Understanding Pods and Logs

A Pod in Kubernetes is the smallest deployable unit and is a logical collection of one or more containers. Each container inside the pod runs an instance of an application. The log output from these containers, such as standard output (stdout) or standard error (stderr), provides critical insights into the running application.

When a pod is terminated, whether due to completion or failure, its logs aren't immediately accessible through standard kubectl logs commands. However, Kubernetes offers several strategies to access these logs post termination.

Methods to Access Logs of Terminated Pods

1. Using Ephemeral Storage

When a pod is terminated, the log information is not retained on the ephemeral storage by default. However, Kubernetes provides a transient solution:

• Restart Policies: If the pod's restart policy is set to Always or OnFailure, logs can be temporarily viewed if a replacement pod with the same name or similar characteristics is induced by a ReplicaSet or Deployment update.
• Job Resources: In scenarios where the workload is part of a Job and the pod completes successfully, logs can be accessed via kubectl logs until the pod is cleaned up by Kubernetes garbage collection.

Command:

kubectl logs <pod-name> --previous

2. Implementing Persistent Storage

Persistent storage solutions offer an ideal fallback for accessing logs after pod termination:

• Persistent Volumes: By configuring a PersistentVolume and PersistentVolumeClaim, you can store logs within a mounted directory. This approach ensures that logs persist beyond the lifecycle of the pods.
• Example Configuration:

1apiVersion: v1
2kind: Pod
3metadata:
4  name: logging-pod
5spec:
6  containers:
7  - name: logger
8    image: busybox
9    command: ["/bin/sh", "-c", "while true; do echo Hello Kubernetes; sleep 10; done"]
10    volumeMounts:
11    - name: log-storage
12      mountPath: /var/log
13  volumes:
14  - name: log-storage
15    persistentVolumeClaim:
16      claimName: log-pvc

3. Utilizing Log Aggregation Solutions

For a more robust logging solution, consider using advanced log aggregation tools. These tools not only centralize logs but provide querying and visualization capabilities:

• ELK Stack (Elasticsearch, Logstash, and Kibana): Aggregates logs and provides a UI for searching and visualizing logs.
• Fluentd: Acts as a unified logging layer, funneling logs from pods to various back-ends.
• Promtail with Loki and Grafana: Transfers logs to the Loki storage back-end, allowing visualization in Grafana.

Example Deployment with Fluentd to capture global logs:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: fluentd-config
5  namespace: kube-system
6data:
7  fluentd.conf: |
8    <source>
9      @type tail
10      path /var/log/containers/*.log
11      pos_file /var/log/td-agent/containers.log.pos
12      tag kubernetes.*
13      format json
14    </source>
15    <match **>
16      @type stdout
17    </match>

4. Container Runtime Log Accessibility

In some scenarios, platform administrators may access logs preserved by the container runtime directly from the nodes. For example, Docker logs can be found in the default /var/log/containers/ directory. However, this approach often requires elevated permissions and node-level access, which may not be practical in managed environments.

Considerations and Best Practices

• Ensure your logging strategy is scalable to handle the log volume generated by your applications.
• Secure logs; sensitive information should be masked or encrypted.
• Implement log retention policies ensuring you conform to data governance standards.
• Use a log aggregation solution when operating at scale for performance and manageability.

Summary and Comparison

The following table summarizes the key methods to access terminated pod logs:

In conclusion, choosing a strategy depends largely on your application's requirements, the scale of deployment, and the security and compliance considerations unique to your environment. By leveraging persistent storage and log aggregation tools, you can efficiently manage and analyze logs from terminated pods, thus maintaining application stability and performance.