Introduction

Deep merging combines nested dictionaries while preserving existing branches instead of replacing entire sections at the top level. This is common in layered configuration systems where environment overrides should change only selected values. A good deep-merge function should be explicit about conflict policy, avoid accidental mutation, and be easy to test.

Shallow Merge vs Deep Merge

A shallow merge replaces values only by top-level keys. That can silently remove nested data.

1base = {
2    "db": {"host": "localhost", "port": 5432, "pool": {"min": 1, "max": 5}},
3    "features": {"search": True},
4}
5
6override = {
7    "db": {"port": 5433},
8}
9
10shallow = base | override
11print(shallow)

In this result, the nested db dictionary from base is replaced entirely. A deep merge should keep unchanged nested keys and only override requested parts.

Basic Recursive Deep Merge

A common strategy is recursion when both values are dictionaries.

1from collections.abc import Mapping
2
3
4def deep_merge(base: dict, incoming: dict) -> dict:
5    result = dict(base)
6
7    for key, value in incoming.items():
8        if key in result and isinstance(result[key], Mapping) and isinstance(value, Mapping):
9            result[key] = deep_merge(result[key], value)
10        else:
11            result[key] = value
12
13    return result
14
15
16left = {
17    "db": {"host": "localhost", "port": 5432, "pool": {"min": 1, "max": 5}},
18    "features": {"search": True},
19}
20right = {
21    "db": {"port": 5433, "pool": {"max": 20}},
22    "features": {"recommendations": True},
23}
24
25print(deep_merge(left, right))

This creates a new dictionary and leaves inputs unchanged.

Define Conflict Policy Explicitly

Deep merge behavior is not universal. Teams should decide how to handle lists, None, and type mismatches.

Example policy choices:

• when both values are lists: replace, append, or append unique
• when incoming value is None: keep existing or treat as explicit clear
• when types differ: override or raise an error

A configurable helper keeps these decisions explicit.

1from collections.abc import Mapping
2from typing import Literal
3
4ListPolicy = Literal["replace", "append", "unique_append"]
5
6
7def deep_merge_policy(base: dict, incoming: dict, list_policy: ListPolicy = "replace") -> dict:
8    result = dict(base)
9
10    for key, value in incoming.items():
11        current = result.get(key)
12
13        if isinstance(current, Mapping) and isinstance(value, Mapping):
14            result[key] = deep_merge_policy(current, value, list_policy)
15        elif isinstance(current, list) and isinstance(value, list):
16            if list_policy == "append":
17                result[key] = current + value
18            elif list_policy == "unique_append":
19                seen = set(current)
20                result[key] = current + [x for x in value if x not in seen]
21            else:
22                result[key] = value
23        else:
24            result[key] = value
25
26    return result

Keeping this policy in one helper avoids hidden behavior differences across services.

Mutation and Copy Strategy

Some deep-merge snippets mutate the base dictionary directly. That can be fine for controlled scripts but risky in request handlers or shared caches. Non-mutating merges are usually easier to reason about because callers can compare original and merged states safely.

If performance is a concern, profile before switching to in-place mutation. For most config workloads, clarity and correctness matter more than micro-optimization.

Testing Deep Merge Logic

Merge helpers look simple but fail in edge cases. Add tests for:

• nested dictionary override
• list policy behavior
• type mismatches
• empty dictionaries
• immutability of inputs

Example quick tests:

assert deep_merge({"a": {"b": 1}}, {"a": {"c": 2}}) == {"a": {"b": 1, "c": 2}}
assert deep_merge({"x": 1}, {"x": 2}) == {"x": 2}

These checks catch regressions when policy logic evolves.

Large Payload Considerations

For large nested payloads, repeated deep merges can be expensive. Practical optimizations:

• merge shared defaults once at startup
• cache merged profiles when input combinations repeat
• avoid unnecessary merging in tight loops

Do not optimize blindly. Measure actual merge cost in realistic workloads first.

Common Pitfalls

• Using shallow merge operators and assuming nested keys are preserved.
• Mutating original dictionaries unintentionally.
• Leaving list behavior undefined across modules.
• Ignoring type mismatches that should trigger explicit errors.
• Skipping tests because recursive merge function looks small.

Summary

• Deep merge preserves nested branches while applying targeted overrides.
• Recurse only when both values are dictionary-like objects.
• Document and implement explicit conflict policies for lists and mismatched types.
• Prefer non-mutating merge results for safer reasoning.
• Add focused tests to keep merge behavior stable over time.