Apache Hadoop MapReduce is a framework for processing large volumes of data across clusters of computers using simple programming models. It utilizes the YARN framework to manage distributed computing resources and uses MapReduce to process data in a series of two-step operations: mapping and reducing. As part of its operational infrastructure, Hadoop uses Apache log4j, a flexible logging framework for Java, to handle logging messages which are crucial for diagnosing and debugging software issues.

By default, Hadoop directs log4j-generated log messages to STDERR and STDOUT files located under the userlogs/job_<jobid>/ directory. Modification of these logging behaviors often becomes necessary to organize logs better, separate them by log levels, or distinguish them by other meaningful partitions. While MapReduce programs inherently offer tremendous computational power, understanding and managing the vast logging they produce is vital for effective application management.

Customizing Log4j Properties in MapReduce

To direct log messages to a custom file in the userlogs/job_<jobid>/ directory, you must configure the log4j properties specific to your MapReduce job. This can be achieved either by setting properties programmatically within your job's code or by using external log4j configuration files.

1. Log4j Configuration through Code

You can configure log4j programmatically by manipulating the Logger object in your MapReduce job code. Here is an illustrative example:

1import org.apache.log4j.Logger;
2import org.apache.log4j.FileAppender;
3import org.apache.log4j.SimpleLayout;
4
5public class MapReduceJob {
6
7  public static void main(String[] args) throws Exception {
8    Logger logger = Logger.getLogger(MapReduceJob.class);
9    FileAppender appender = new FileAppender(new SimpleLayout(), "custom_log.log");
10    logger.addAppender(appender);
11
12    // Rest of your MapReduce code
13  }
14}

This code snippet will create and use a custom_log.log in the working directory. However, to ensure the log file is placed specifically under the userlogs/job_<jobid>/ directory, you might need additional configuration or command-line options when launching jobs.

2. External Log4j Configuration

A more maintainable and flexible approach is to use an external log4j configuration file. This allows changes without modifying the job's source code. You can specify the log4j configuration file using the -files option of the hadoop jar command.

hadoop jar your-job.jar -files /path/to/log4j.properties

Your log4j.properties should redirect the logging to the desired file:

1log4j.rootLogger=INFO, file
2log4j.appender.file=org.apache.log4j.FileAppender
3log4j.appender.file.File=/path/to/custom_log.log
4log4j.appender.file.layout=org.apache.log4j.PatternLayout
5log4j.appender.file.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

Important Considerations

• Environmental Impact: Note that the file paths may require dynamic behavior to adapt to different YARN containers' paths.
• Performance: Excessive logging, especially to disk, can significantly impact the performance of your MapReduce jobs.

Summary Table

Conclusion

Effective logging is critical for monitoring and troubleshooting in Hadoop MapReduce environments. Whether through programmatic configurations within the job itself or by external configuration files, managing logging location and level can greatly enhance the operational management of MapReduce jobs. Always consider the environmental impact and performance implications when customizing the logging infrastructure.