Key Management and Envelope Encryption

Topics Covered

Why Key Management Is the Hard Part

Why keys are the real problem

Where keys go wrong

The key lifecycle

Key hierarchies

A hierarchy in numbers

Separation of duties

Purpose separation

Key derivation

Randomness and generation

Who should hold keys

Crypto shredding

Questions to settle before encrypting

Strong algorithms are the easy part of cryptography. The hard part is keys: generating them, storing them, controlling who can use them, rotating them, backing them up and destroying them. Almost every real world encryption failure is a key management failure, such as a key committed to a repository, a key stored next to the data it protects or a key that no one can rotate because nobody knows what it encrypts. This lesson covers the key lifecycle, envelope encryption, cloud key management services, rotation, access control and the operational habits that keep encrypted data both safe and recoverable.

Why keys are the real problem

Encryption converts a large secret, the data, into a small secret, the key. That is useful because a small secret is easier to protect, but it means the security of every encrypted byte now rests on the key. If the key leaks, encryption provides nothing. If the key is lost, the data is gone for good. If too many people or services can use the key, encryption adds a step but not a barrier. Good key management is about making the small secret genuinely easier to protect than the data it guards.

Where keys go wrong

Reviews of real systems find the same mistakes again and again. Keys are hardcoded in source code or configuration files and committed to version control. Keys are placed in environment variables that end up in crash reports and debug pages. The same key is used for every customer, every environment and every purpose. Keys are stored in the same database as the encrypted data, so one backup leak exposes both. Keys are never rotated because the application has no way to tell which key encrypted which record. Each mistake turns encryption into decoration.

Of 120 services reviewed, 72 kept encryption keys in code, environment variables or beside the data. Only 48 used a KMS or secrets manager.

The key lifecycle

Every key passes through the same stages. It is generated with a strong random source, ideally inside the system that will hold it. It is stored and distributed to the components that need it, with access controls. It is used for a defined purpose, and its use is logged. It is rotated, meaning new data is protected with a new key while the old one remains available for decryption. It may be suspended or revoked if compromise is suspected. Finally it is destroyed once nothing depends on it. Each stage needs an owner and a documented procedure, written before the first key is created.

Key hierarchies

Real systems rarely use one key directly. They use a hierarchy. At the top sits a root or master key, held in a hardware security module or a cloud key management service and almost never used directly. Below it are key encryption keys, which encrypt other keys. At the bottom are data encryption keys, which encrypt the actual data. Each layer protects the layer below. The hierarchy lets a single well guarded root protect millions of data keys, and lets the system rotate or revoke at whatever level matches the event, from one record to a whole tenant.

A hierarchy in numbers

The scale of a hierarchy is easy to underestimate. A SaaS product with four hundred tenants might have one root key in the KMS, one key encryption key per tenant and a fresh data key for every stored object, adding up to tens of millions of data keys. Only the root and the tenant keys are managed objects with policies. The data keys are small wrapped blobs stored beside their objects. When the team needs to cut off one tenant, it disables that tenant key, and every object for that tenant becomes unreadable at once without touching any other tenant.

One KMS root protects 400 tenant keys, which wrap about 38 million data keys. Disabling one tenant key cuts off about 95,000 objects without touching any other tenant.

Separation of duties

Key management works best when no single person or service can both use a key and change who may use it. Key administrators, who create keys and set policies, should be different from key users, which are usually application roles that can encrypt and decrypt but cannot change policies or delete keys. Cloud key services support this split directly in their permission models. The aim is that a single compromised account, whether an engineer laptop or an application server, cannot both read data and hide the evidence by changing policies or destroying audit trails.

Purpose separation

A key should serve one purpose. A key that encrypts payment data should not also sign session cookies, and a key used in production should never be used in staging. Using one key across purposes multiplies the impact of a leak, makes rotation risky because more things depend on it and can create cryptographic interactions where output from one use helps attack another. Separate keys per purpose, per environment and often per tenant cost very little with a key service, and they make every later decision about rotation, access and incident response simpler.

Key derivation

Sometimes one secret needs to become many keys, for example a separate key per tenant or per purpose from a single root. A key derivation function such as HKDF does this safely: it takes the root key and a context string, such as the tenant identifier and purpose, and produces an independent looking key for each context. Derived keys do not need to be stored, since they can be recomputed, and compromise of one derived key does not reveal the root or other derived keys. Derivation is not a substitute for a key service but often complements one.

Randomness and generation

Keys must come from a cryptographically secure random source. Operating system generators, such as the getrandom system call on Linux or the secrets module in Python, are suitable. General purpose random functions intended for simulations and games are not, because their output can be predicted from a few observed values. Keys derived from passwords need a slow password based derivation function, as covered in the hashing lesson, because human chosen passwords have little entropy. Generating keys inside a key management service removes the question entirely, which is one more reason to prefer it.

Who should hold keys

Teams often ask whether the cloud provider, the application or a separate system should hold keys. The answer depends on the attacker. Provider managed keys protect against theft of physical disks and many provider side mistakes. Customer managed keys in the provider key service add control over policy, auditing and destruction. Keys held outside the provider, in an external key store or the application, protect against the provider itself but bring significant operational burden. Most teams should use customer managed keys in their cloud KMS, and reserve external key control for specific regulatory or contractual requirements.

Crypto shredding

Key management also offers a powerful way to delete data. If each customer or each dataset is encrypted under its own key, destroying the key makes every copy of that data unreadable, including copies in backups, replicas and logs that would otherwise be impractical to find. This technique, called crypto shredding, is used to honour deletion requests and to retire datasets cleanly. It only works if keys are scoped finely enough and if no copies of the key survive elsewhere, which the key service and its backups must guarantee.

Questions to settle before encrypting

Before any code is written, a team should be able to say which attacker the encryption is meant to stop, where the key will live relative to that attacker, who may use the key and how that is enforced, how keys will be rotated and how data will be deleted. These questions take an hour to answer and often reveal that the planned encryption would not stop the attacker of concern, for example because the key would sit on the same host as the data, which is far better to learn before launch than after an incident.

Protect the small secret

Treat key management as the core of any encryption design. Use a hierarchy with a root key in a KMS or HSM, separate keys by purpose, environment and tenant, split key administration from key use, generate keys with secure randomness, document each lifecycle stage and remember that destroying a key is a reliable way to destroy data.