Practice Now
Hadoop
Map/Reduce
DistributedCache
Big Data
Technology

How do I access DistributedCache in Hadoop Map/Reduce jobs?

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Big Data

Apache Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. One of the key components of Hadoop is the Map/Reduce job model, which splits the input data into independent chunks processed by the map tasks in a completely parallel manner. The intermediate output is then processed by reduce tasks to produce the final result.

DistributedCache is a facility provided by the Apache Hadoop framework to cache files (text, archives, jars, etc.) needed by applications. Once you cache a file for your Hadoop job, Hadoop makes it available on each data node where your map/reduce tasks are running, providing a significant performance boost.

Why Use DistributedCache?

  1. Efficiency: Caching files on each node rather than each task fetching them from a central location saves bandwidth and reduces network congestion.
  2. Speed: Access to local files is significantly faster than pulling them over the network.
  3. Convenience: It abstracts file management from the user, simplifying the coding process.

How to Use DistributedCache in Map/Reduce Jobs

To leverage DistributedCache, you generally follow these steps:

Adding Files to the DistributedCache

  1. Identify the files you want to share across all nodes.
  2. Add these files to the cache at the time of job configuration:
java
   Job job = new Job(config, "ExampleJob");
   DistributedCache.addCacheFile(new URI("/path/to/your/file.txt#alias"), job.getConfiguration());

Here, #alias is an optional way to provide an alias by which you will reference the file in the Map/Reduce code. If an alias is used, the file will appear in the task working directory with the given alias.

Accessing Cached Files in a Map/Reduce Program

  1. In the Mapper or Reducer setup method: Access the file through the local file system API. The file is located in the directory where the task runner is executed.
java
1   Path[] cacheFiles = DistributedCache.getLocalCacheFiles(job.getConfiguration());
2   if (cacheFiles != null && cacheFiles.length > 0) {
3     for (Path cachePath : cacheFiles) {
4       // Read your file as needed using regular File I/O operations
5     }
6   }

Practical Example:

Suppose your Map/Reduce job requires a lookup file called lookup.dat stored in HDFS, and you want to use this file in your map tasks. Below is how you would add it to your setup:

java
1public class CacheExample extends Configured implements Tool {
2
3    @Override
4    public int run(String[] args) throws Exception {
5        Configuration conf = getConf();
6        Job job = Job.getInstance(conf, "Cache Example");
7
8        job.setJarByClass(CacheExample.class);
9        job.setMapperClass(MyMapper.class);
10        job.setNumReduceTasks(0);
11
12        FileInputFormat.addInputPath(job, new Path(args[0]));
13        FileOutputFormat.setOutputPath(job, new Path(args[1]));
14
15        // Add files to DistributedCache
16        DistributedCache.addCacheFile(new URI("/path/to/lookup.dat"), job.getConfiguration());
17
18        return job.waitForCompletion(true) ? 0 : 1;
19    }
20
21    public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
22
23        private Map lookupData = new HashMap();
24
25        @Override
26        public void setup(Context context) throws IOException {
27            Path[] cachedFiles = DistributedCache.getLocalCacheFiles(context.getConfiguration());
28            if (cachedFiles != null && cachedFiles.length > 0) {
29                BufferedReader reader = new BufferedReader(new FileReader(cachedFiles[0].toString()));
30                String line;
31                while ((line = reader.readLine()) != null) {
32                    // Process the lookup data
33                }
34                reader.close();
35            }
36        }
37
38        @Override
39        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
40            // Map logic here using lookupData
41        }
42    }
43
44    public static void main(String[] args) throws Exception {
45        int res = ToolRunner.run(new Configuration(), new CacheExample(), args);
46        System.exit(res);
47    }
48}

Summary Table

FeatureDescription
UtilityProvides a mechanism to cache files required by your Hadoop job across all nodes.
Performance BoostReduces network congestion and saves bandwidth by sharing files locally rather than over the network.
Ease of UseSimplifies job configuration by abstracting complex file management tasks.
Integration PointImplements through the Job configuration and accessible within the Map and Reduce tasks.
AccessibilityCached files can be accessed as local files within the task runs.

By efficiently using DistributedCache, developers can enhance the performance of their Hadoop jobs and manage data sharing in a more effective and controlled manner.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.