Introduction
Python's built-in dict.update() only merges at the top level — nested dictionaries are replaced, not merged. To recursively update a nested dictionary of arbitrary depth, you need a custom function that walks both dictionaries and merges at every level. This is essential when merging configuration files, API responses, or any hierarchical data where you want to preserve existing nested values while updating specific keys.
The Problem with dict.update()
1default_config = {
2 "database": {
3 "host": "localhost",
4 "port": 5432,
5 "credentials": {
6 "user": "admin",
7 "password": "secret"
8 }
9 },
10 "logging": {
11 "level": "INFO"
12 }
13}
14
15override = {
16 "database": {
17 "credentials": {
18 "password": "new_password"
19 }
20 }
21}
22
23# dict.update replaces the entire "database" key
24default_config.update(override)
25print(default_config)
26# {'database': {'credentials': {'password': 'new_password'}}, 'logging': {'level': 'INFO'}}
27# host, port, and user are GONE
update() replaced the entire "database" dictionary with the override, losing host, port, and user.
Recursive Deep Merge
1def deep_update(base, override):
2 """Recursively update base dict with override dict."""
3 for key, value in override.items():
4 if key in base and isinstance(base[key], dict) and isinstance(value, dict):
5 deep_update(base[key], value)
6 else:
7 base[key] = value
8 return base
9
10# Usage
11default_config = {
12 "database": {
13 "host": "localhost",
14 "port": 5432,
15 "credentials": {"user": "admin", "password": "secret"}
16 },
17 "logging": {"level": "INFO"}
18}
19
20override = {
21 "database": {
22 "credentials": {"password": "new_password"}
23 }
24}
25
26result = deep_update(default_config, override)
27print(result)
28# {
29# 'database': {
30# 'host': 'localhost',
31# 'port': 5432,
32# 'credentials': {'user': 'admin', 'password': 'new_password'}
33# },
34# 'logging': {'level': 'INFO'}
35# }
Only the password key was updated. All other nested values were preserved.
Non-Mutating Version
The function above modifies base in place. To preserve the original:
1import copy
2
3def deep_merge(base, override):
4 """Return a new dict with base deeply merged with override."""
5 result = copy.deepcopy(base)
6 for key, value in override.items():
7 if key in result and isinstance(result[key], dict) and isinstance(value, dict):
8 result[key] = deep_merge(result[key], value)
9 else:
10 result[key] = copy.deepcopy(value)
11 return result
12
13merged = deep_merge(default_config, override)
14# default_config is unchanged
Setting a Value at an Arbitrary Path
Update a specific key path like ["database", "credentials", "password"]:
1def set_nested(d, keys, value):
2 """Set a value in a nested dict using a list of keys."""
3 for key in keys[:-1]:
4 d = d.setdefault(key, {})
5 d[keys[-1]] = value
6
7config = {}
8set_nested(config, ["database", "credentials", "password"], "secret123")
9print(config)
10# {'database': {'credentials': {'password': 'secret123'}}}
setdefault creates intermediate dictionaries if they do not exist.
Getting a Value at an Arbitrary Path
1def get_nested(d, keys, default=None):
2 """Get a value from a nested dict using a list of keys."""
3 for key in keys:
4 if isinstance(d, dict):
5 d = d.get(key, default)
6 else:
7 return default
8 return d
9
10config = {"database": {"credentials": {"password": "secret"}}}
11print(get_nested(config, ["database", "credentials", "password"]))
12# secret
13print(get_nested(config, ["database", "missing", "key"], "fallback"))
14# fallback
Deleting a Nested Key
1def delete_nested(d, keys):
2 """Delete a key from a nested dict."""
3 for key in keys[:-1]:
4 d = d.get(key, {})
5 if not isinstance(d, dict):
6 return # Path doesn't exist
7 d.pop(keys[-1], None)
8
9config = {"database": {"credentials": {"user": "admin", "password": "secret"}}}
10delete_nested(config, ["database", "credentials", "password"])
11print(config)
12# {'database': {'credentials': {'user': 'admin'}}}
Using Libraries
For production code, use established libraries:
1# deepmerge library
2from deepmerge import always_merger
3
4result = always_merger.merge(base_dict, override_dict)
5
6# pydantic-settings for config merging
7# merges environment variables, .env files, and defaults automatically
8
9# python-box for dot-access nested dicts
10from box import Box
11
12config = Box({"database": {"host": "localhost"}})
13config.database.host = "production-server"
Merging Multiple Dictionaries
1from functools import reduce
2
3def deep_merge_all(*dicts):
4 """Deep merge multiple dictionaries left to right."""
5 return reduce(deep_merge, dicts)
6
7defaults = {"a": {"x": 1, "y": 2}}
8env_config = {"a": {"y": 3, "z": 4}}
9cli_args = {"a": {"z": 5}, "b": 10}
10
11final = deep_merge_all(defaults, env_config, cli_args)
12print(final)
13# {'a': {'x': 1, 'y': 3, 'z': 5}, 'b': 10}
Common Pitfalls
Using dict.update() for nested merging: update() replaces nested dictionaries entirely instead of merging them. Always use a recursive function for nested data.
Modifying the original dictionary: The in-place deep_update function mutates base. If you need to preserve the original, use copy.deepcopy() first or use the non-mutating deep_merge version.
Infinite recursion with circular references: If a dictionary contains references to itself (e.g., d["self"] = d), recursive merge loops forever. Add a depth limit or a visited-set guard for untrusted input.
Non-dict values at merge points: If base["key"] is a string but override["key"] is a dict (or vice versa), the recursive check isinstance(base[key], dict) and isinstance(value, dict) correctly replaces instead of merging. Ensure this behavior matches your expectations.
List merging: The recursive function above replaces lists entirely. If you need to merge lists (append, deduplicate, or zip), add explicit list handling: if isinstance(base[key], list) and isinstance(value, list): base[key].extend(value).
Summary
dict.update() only merges the top level — nested dicts are replaced, not merged
Use a recursive deep_update function to merge at every nesting level
Use copy.deepcopy() for a non-mutating merge that preserves the original
Use setdefault in a loop to set values at arbitrary key paths
For production code, consider libraries like deepmerge or python-box
Handle edge cases: circular references, type mismatches at merge points, and list values
