Apache Hadoop’s MapReduce is a software framework for easily writing applications that process vast amounts of data (multi-terabyte datasets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner. A typical MapReduce job often needs additional files or data for processing such as a serialized object, configuration XML, etc. Hadoop provides several mechanisms to make these external files available to tasks in the MapReduce jobs.

Using the Distributed Cache in Hadoop MapReduce

One effective way to add external files into a MapReduce application is the use of the Distributed Cache functionality, which is designed to cache files needed by applications. Once you add a file to the Distributed Cache, Hadoop framework makes it available on each data node where map/reduce tasks are running, thus eliminating the need to fetch it multiple times. This is particularly useful for sharing large datasets or configuration files across all nodes.

Steps to Add Files in Distributed Cache:

1. During Job Configuration: Before submitting the job, specify the files that need to be shared via the Distributed Cache using Job Configuration.

1    import org.apache.hadoop.filecache.DistributedCache;
2    import org.apache.hadoop.mapreduce.Job;
3
4    public class DriverClass {
5        public static void main(String[] args) throws Exception {
6            Job job = Job.getInstance(new Configuration());
7
8            // Adding files to Distributed Cache
9            DistributedCache.addCacheFile(new URI("/path/to/your/file#alias"), job.getConfiguration());
10            
11            // Continue with setting job input format, map class etc.
12        }
13    }

Here, files can be added to the Distributed Cache, and an alias can be optionally provided so that files can be referenced in the Map and Reduce jobs.

1. Accessing Files in Map or Reduce Tasks: Once the job is launched, you can access these cached files in your Mapper or Reducer tasks.

1    import org.apache.hadoop.mapreduce.Mapper;
2    import java.net.URI;
3    import org.apache.hadoop.fs.Path;
4
5    public class MyMapper extends Mapper<...> {
6        protected void setup(Context context) throws IOException, InterruptedException {
7            Path[] cachedFiles = DistributedCache.getLocalCacheFiles(context.getConfiguration());
8            if (cachedFiles != null && cachedFiles.length > 0) {
9                for (Path cachedFile : cachedFiles) {
10                    // Read the file from cachedFile using IO operations
11                }
12            }
13        }
14    }

Options for Managing Multiple Files or Complex Dependencies

When dealing with multiple dependencies or complex file structures like entire directories, the above approach of adding each file to the Distributed Cache individually might not be efficient. In such cases, you might consider compressing your files or directories into a single archive.

Adding an Archive to Distributed Cache:

You can add compressed archives (e.g., tar.gz, zip) to the Distributed Cache, and Hadoop will uncompress these archives at each data node.

DistributedCache.addCacheArchive(new URI("/path/to/your/archive.tar.gz#aliasForArchive"), job.getConfiguration());

Handling Updates to Cached Files

Caching is indeed efficient; however, when files are updated frequently, you need a strategy to refresh cache items. One approach can be versioning files and ensuring your job configuration links to the correct version.

Conclusion and Summary

Adding external files in MapReduce can significantly enhance the efficiency and flexibility of your jobs. Various Hadoop-native mechanisms, such as the Distributed Cache, provide a robust solution for dealing with such requirements, whether it's small configuration files or large binary datasets.

Here's a summary of key procedures and their explanations:

Understanding and utilizing Hadoop’s built-in features like Distributed Cache can significantly reduce the complexity and runtime of MapReduce jobs dealing with external datasets or dependencies.