How to get Filename/File Contents as key/value input for MAP when running a Hadoop MapReduce Job?

Hadoop

MapReduce

Big Data

Data Processing

File Handling

How to get Filename/File Contents as key/value input for MAP when running a Hadoop MapReduce Job?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

In the context of Hadoop MapReduce, processing files and their contents efficiently as key/value pairs is crucial for many big data applications. This article will elaborate on how to get the filename and file contents as key/value input for a MapReduce job, utilizing the Hadoop API, and will also include a practical example.

Understanding Hadoop MapReduce InputFormats

Hadoop MapReduce uses InputFormats to define how input files are split and read. Each InputFormat provides implementations for:

InputSplit: Logical representation of data split.
RecordReader: Process to read the input split data into key/value pairs.

The default InputFormat in Hadoop is TextInputFormat, which treats each line of input files as a separate record. It assigns the byte offset as the key and the line content as the value, and does not provide the file name in the key. To include the file name, you must use or customize an InputFormat.

Customizing InputFormat to Include Filename

To retrieve both the filename and its contents as key/value pairs, you will have to implement a custom InputFormat. Here's how to do it:

1. Custom InputFormat Class

Create a custom class that extends FileInputFormat<KEYOUT, VALUEOUT>, overriding the createRecordReader method. This method should return a new instance of your custom RecordReader.

java

1public class FileNameInputFormat extends FileInputFormat<Text, Text> {
2    @Override
3    protected boolean isSplitable(JobContext context, Path file) {
4        return false;
5    }
6
7    @Override
8    public RecordReader<Text, Text> createRecordReader(
9            InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException {
10        return new FileNameRecordReader();
11    }
12}

2. Custom RecordReader Class

Develop a RecordReader class that can read the input splits and process files to emit filename as key and content as value.

java

1public class FileNameRecordReader extends RecordReader<Text, Text> {
2
3    private Text key = new Text();
4    private Text value = new Text();
5    private LineReader in;
6    private long start;
7    private long end;
8    private long pos;
9    private FSDataInputStream fileIn;
10
11    @Override
12    public void initialize(InputSplit genericSplit, TaskAttemptContext context)
13            throws IOException, InterruptedException {
14        FileSplit split = (FileSplit) genericSplit;
15        Configuration job = context.getConfiguration();
16        final Path file = split.getPath();
17        FileSystem fs = file.getFileSystem(job);
18
19        fileIn = fs.open(split.getPath());
20        in = new LineReader(fileIn, job);
21        this.start = split.getStart();
22        this.end = start + split.getLength();
23        this.pos = start;
24        this.key.set(file.getName());
25    }
26
27    @Override
28    public boolean nextKeyValue() throws IOException {
29        if (pos < end) {
30            int newSize = in.readLine(value);
31            pos += newSize;
32            return newSize != 0;
33        }
34        return false;
35    }
36
37    @Override
38    public Text getCurrentKey() {
39        return key;
40    }
41
42    @Override
43    public Text getCurrentValue() {
44        return value;
45    }
46
47    @Override
48    public float getProgress() throws IOException {
49        if (start == end) {
50            return 0.0f;
51        } else {
52            return Math.min(1.0f, (pos - start) / (float)(end - start));
53        }
54    }
55
56    @Override
57    public void close() throws IOException {
58        if (in != null) {
59            in.close();
60        }
61        if (fileIn != null) {
62            fileIn.close();
63        }
64    }
65}

3. Configuring Job to Use Custom InputFormat

In your MapReduce driver code, set the InputFormat class to your custom class:

java

Job job = Job.getInstance(new Configuration());
job.setInputFormatClass(FileNameInputFormat.class);

Key Benefits and Points

Feature	Description
File as Key	Easily identify the source file in map/reduce tasks.
Customizability	Flexibility in processing file content beyond line-wise.
Efficiency	Process each file only once without redundant I/O.
Scalability	Suitable for expanding datasets and larger applications.

Conclusion

Implementing a custom InputFormat and RecordReader for processing filenames along with their contents as key/value pairs can be highly beneficial. This approach allows for more effective data processing pipelines, providing context (file metadata) alongside data (content), indispensable for many analytical and data transformation tasks in big data platforms like Hadoop.