Introduction

Hadoop's old DistributedCache API was designed to copy small read-only files to task nodes so mappers and reducers could use shared reference data locally. In modern Hadoop code, you usually do not call the old DistributedCache class directly anymore. Instead, you add cache files through the Job API and let YARN localize them for the task.

Know the Modern API Path

Many examples still mention DistributedCache, but the modern pattern uses job.addCacheFile(...) or job.addCacheArchive(...).

1import java.net.URI;
2import org.apache.hadoop.conf.Configuration;
3import org.apache.hadoop.mapreduce.Job;
4
5Configuration conf = new Configuration();
6Job job = Job.getInstance(conf, "lookup-job");
7job.addCacheFile(new URI("hdfs:///data/lookup/country_codes.csv#country_codes.csv"));

The fragment after # creates a symlink name inside the task's working directory. That makes the localized file easier to open.

This is the practical replacement for the older DistributedCache.addCacheFile(...) style.

Access the Localized File in the Mapper or Reducer

Cached files are typically read during setup, not for every individual record.

1import java.io.BufferedReader;
2import java.io.FileReader;
3import java.io.IOException;
4import java.util.HashMap;
5import java.util.Map;
6import org.apache.hadoop.io.LongWritable;
7import org.apache.hadoop.io.Text;
8import org.apache.hadoop.mapreduce.Mapper;
9
10public class CountryMapper extends Mapper<LongWritable, Text, Text, Text> {
11    private final Map<String, String> countryMap = new HashMap<>();
12
13    @Override
14    protected void setup(Context context) throws IOException {
15        try (BufferedReader reader = new BufferedReader(new FileReader("country_codes.csv"))) {
16            String line;
17            while ((line = reader.readLine()) != null) {
18                String[] parts = line.split(",", 2);
19                if (parts.length == 2) {
20                    countryMap.put(parts[0], parts[1]);
21                }
22            }
23        }
24    }
25}

This pattern keeps lookup data in memory after one local read, which is the normal reason for using the cache in the first place.

Use It for Small, Read-Only Reference Data

The cache is best for things such as:

• lookup tables
• static configuration files
• dictionaries
• archived helper resources needed by every task

It is not for large mutable datasets. If the file is large enough to dominate startup or localization cost, it may belong in ordinary HDFS reads or a different data distribution design.

The important assumption is that every task sees the same read-only file content.

Archives and Executables Need Slightly Different Handling

If you need to distribute a zip or tar archive, use a cache archive so Hadoop localizes and unpacks it.

job.addCacheArchive(new URI("hdfs:///tools/my-tool.zip#my-tool"));

Then the unpacked directory is accessible locally through the alias name. This is useful for shipping auxiliary binaries or structured reference resources, but you should keep them small and deterministic.

Prefer Explicit HDFS Paths and Stable Aliases

Use clear HDFS URIs and stable alias names. Debugging gets harder when the localized filename is unpredictable or when code assumes a path that was never aliased.

A good pattern is:

• put the reference file in HDFS first
• add it with a clear alias
• read it in setup
• fail fast if the file is missing or malformed

That makes task behavior repeatable across nodes.

Remember That the Old API Name Is Mostly Historical Now

If you search older blog posts, you will see direct references to DistributedCache. The concept still exists, but the recommended coding style has moved into the Job API. When maintaining older code, do not be surprised if both styles appear. The operational idea is the same: localize shared read-only resources for each task.

Common Pitfalls

• Using the deprecated old DistributedCache API style when the Job API already provides the modern entry points.
• Reading the localized file for every record instead of loading it once in setup.
• Putting large or frequently changing datasets into the cache when it is meant for small read-only resources.
• Forgetting the alias fragment and then opening the wrong local filename in the task.
• Assuming cache localization replaces proper HDFS input design for ordinary large data access.

Summary

• In modern Hadoop, add cached resources through job.addCacheFile or job.addCacheArchive.
• Use the cache for small read-only files that every task needs locally.
• Load the localized resource once during task setup.
• Use clear HDFS URIs and stable alias names for predictable access.
• Treat DistributedCache as a concept that survives, even though the older class-based API is mostly legacy.