Capture Kubernetes Spark driver and executor logs in S3 and view in History Server
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
For Spark on Kubernetes, there are two different kinds of logs people often mix together: Spark event logs and raw pod logs. The Spark History Server reads event logs, not Kubernetes stdout and stderr logs, so the solution is to send event logs to S3 for History Server and use a separate log collector if you also want raw driver and executor container logs in S3.
Send Spark Event Logs to S3
To make completed applications appear in the History Server, enable event logging and point it at an S3 path that both the Spark jobs and the History Server can read.
This stores Spark event logs in S3. Those logs contain job, stage, task, SQL, and executor event metadata that the History Server replays to reconstruct the UI.
Configure the History Server to Read the Same S3 Path
The History Server must point at the same directory through spark.history.fs.logDirectory.
If the History Server runs in Kubernetes, it also needs the right Hadoop AWS libraries, S3 configuration, and credentials available in the pod. Once that is set, the History Server UI can list completed applications whose event logs exist in S3.
Raw Driver and Executor Logs Are Different
kubectl logs shows pod logs from the driver and executor containers. Those are useful for stack traces, print statements, and shell-level failures, but the History Server does not ingest them directly.
That means this goal actually splits into two tasks:
- event logs to S3 for Spark History Server
- container logs to S3 through a Kubernetes log pipeline
If you want raw pod logs in S3, use a collector such as Fluent Bit, Fluentd, or another cluster logging solution that tails container logs and ships them to S3 or CloudWatch.
Example Fluent-Bit Style Architecture
A common pattern is:
- Spark writes event logs to
s3a://my-bucket/spark-events/ - Kubernetes logging agents ship container stdout and stderr elsewhere
- the History Server reads only the event-log directory
This is an important design boundary. If you expect raw executor logs to appear magically inside the History Server, the configuration will never fully work because the History Server is not a Kubernetes log viewer.
Optional UI Links to External Logs
If you have an external log system, Spark can expose custom log URLs in the UI with executor log link configuration. That can make the History Server page point to your real driver and executor logs, but it still does not mean the History Server stores or parses those raw logs itself.
So the realistic answer is not "put raw logs in S3 and History Server will show them." The realistic answer is "put event logs in S3 for History Server, and point executor log links at your external log store if needed."
Common Pitfalls
- Expecting raw Kubernetes pod logs to be readable by Spark History Server.
- Forgetting to enable
spark.eventLog.enabled, which leaves the History Server with nothing to replay. - Writing event logs to S3 while the History Server points at a different directory.
- Configuring
s3a://paths without the Hadoop AWS libraries and credentials needed by both Spark and the History Server. - Treating
kubectl logsoutput and Spark event logs as though they were the same artifact.
Summary
- Spark History Server reads Spark event logs, not raw driver or executor pod logs.
- Send event logs to S3 with
spark.eventLog.enabled=trueandspark.eventLog.dir=s3a://.... - Point
spark.history.fs.logDirectoryat the same S3 location. - Use a Kubernetes log collector if you also want raw container logs in S3.
- If needed, expose external log links from Spark UI instead of expecting History Server to ingest pod logs directly.
