Monitoring and analyzing metrics during the execution of Spark applications is crucial for optimizing performance and managing resources effectively. Metrics such as output size and records written are of particular importance, providing insights into the data processing details. Thankfully, Spark UI offers a comprehensive interface for accessing these valuable metrics. This article details how to retrieve such metrics from Spark UI, providing technical explanations and real-world examples where applicable.

Understanding Spark UI's Structure

Apache Spark UI is structured into several tabs that provide various insights into the application:

• Jobs Tab: Displays the list of all Spark jobs in the application.
• Stages Tab: Offers detailed insights into individual stages of a Spark job.
• Storage Tab: Shows information about RDDs and DataFrames being cached.
• Environment Tab: Details Spark runtime environment including properties and class paths.
• Executors Tab: Displays information about active and dead executors, along with related metrics.

Retrieving Metrics: Output Size and Records Written

Accessing the Stages Tab

The Stages tab in Spark UI is your go-to location for gathering detailed metrics about output size and records written. Here's how you can navigate and interpret the relevant information:

1. Navigate to Spark UI:
   • When running your Spark application, note the web UI hyperlink provided in the console (by default on port 4040). Open this in your web browser.
2. Select the Stages Tab:
   • Click on the "Stages" tab. This displays all stages related to your Spark jobs along with their metrics.
3. Identify Target Stage:
   • Locate the stage you are interested in, focusing on either its ID or description for ease of navigation.
4. Explore Task Details:
   • Click on the link corresponding to the stage ID. This will take you to a page that provides detailed task-level information within the stage.

Reviewing Output Size and Records Written

Once inside the stage details:

1. Look for the "Tasks" Table:
   • The tasks table will illuminate aspects of each individual task executed as part of the stage.
2. Examine Key Metrics Columns:
   • Output Size: Look for a column labeled "Output Size" which reflects the size of data output from each task.
   • Shuffle Write (or Records Written): Depending on the operation, metrics related to data shuffle such as "Shuffle Write Metrics" can offer insights into records written by tasks.

1| Metric | Description | Location |
2| -------- | ------------- | ---------- |
3| Output Size | Size of data written as output by a task | Stages Tab 
→ Stage Details 
→ Tasks Table |
4| Records Written | Number of records outputted during shuffle or write operations | Stages Tab 
→ Stage Details 
→ Tasks Table | ``` |
5
6### Example
7
8Suppose you have executed a simple DataFrame transformation using Spark:
9
10```scala
11val df = spark.read.csv("data.csv")
12val result = df.filter($"col1" > 100).select($"col2", $"col3")
13result.write.mode("overwrite").csv("output/")

• After execution, navigate to Spark UI.
• Within the Stages tab, observe the stage corresponding to the writing task.
• Explore the details to find output size and number of records written by viewing shuffle write statistics.

Additional Considerations

Event Logs

For post-analysis, Spark also writes event logs containing similar metrics. These logs can be parsed using the spark.eventLog.enabled parameter set to true. This setup allows offline metric analysis using a script or a separate tool.

Utilizing Spark Listeners

Custom metrics collection can be achieved using Spark Listeners in your Spark application:

1import org.apache.spark.scheduler._
2
3class CustomSparkListener extends SparkListener {
4  override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
5    val os = taskEnd.taskMetrics.outputMetrics.bytesWritten
6    val rw = taskEnd.taskMetrics.shuffleWriteMetrics.recordsWritten
7    println(s"Task Output Size: $os, Records Written: $rw")
8  }
9}
10
11spark.sparkContext.addSparkListener(new CustomSparkListener())

Summary Table: Key Points

By leveraging the capabilities of the Spark UI and employing auxiliary methods like event logs and Spark Listeners, you can successfully retrieve and analyze performance metrics including output size and records written, thus allowing for optimized data processing workflows.