Kinesis and Managed Kafka

Topics Covered

Kinesis Data Streams

Shards and partition keys

Capacity modes

Hot shards

Resharding

Producers

Queues deliver each message to one consumer and forget it once it is deleted. Streams keep an ordered log of records for a retention period, and any number of consumers read it independently, each tracking its own position. That difference makes streams the right tool for clickstreams, telemetry, change data capture and event sourcing, where several systems need the same ordered data, some need to replay it, and throughput reaches hundreds of thousands of records per second. AWS offers two main streaming services, Kinesis Data Streams and Amazon MSK, its managed Kafka, and this lesson covers both, starting with Kinesis.

Shards and partition keys

A Kinesis data stream is made of shards. Each shard is an ordered sequence of records with a fixed capacity: writes of up to 1 MB per second or 1,000 records per second, and reads of up to 2 MB per second shared among standard consumers. A producer writes a record with a partition key, Kinesis hashes the key with MD5 into a 128 bit space, and each shard owns a range of that space. All records with the same partition key land in the same shard, in the order they were written, which is how Kinesis provides per key ordering.

Each record gets a sequence number that increases within its shard, and consumers read with a shard iterator from a position: the oldest record, the latest, a specific sequence number, or a timestamp. Records stay in the stream for the retention period, 24 hours by default and up to 365 days with extended retention, and reading does not remove them. A consumer that falls behind can catch up, a new consumer can start from the oldest record, and a consumer with a bug can be rewound to reprocess a time range, all without affecting other consumers.

A useful mental model is that a shard is a small, fixed size pipe with its own ordering, and a stream is a bundle of those pipes behind a hash function. Everything else about Kinesis follows from that picture. Throughput limits apply per pipe, ordering holds per pipe, and consumers read per pipe. When teams reason about a stream as one large pipe, they are surprised by throttling on a single shard, by ordering that holds only per key, and by consumers whose progress differs from shard to shard. Keeping the shard level picture in mind makes those behaviours predictable rather than mysterious.

Capacity modes

Streams run in one of two capacity modes. In provisioned mode, the team chooses the number of shards and pays per shard hour, and must reshard as traffic changes. In on demand mode, Kinesis manages shards automatically, scaling to double the previous peak write throughput within minutes, and charges per gigabyte written and read plus a stream hour charge. On demand suits new or unpredictable workloads and removes resharding work, while provisioned mode is cheaper for steady, well understood traffic, often by a factor of two or more at high volume.

Switching between modes is possible, with limits on how often, so a common pattern is to start a new stream in on demand mode, observe a few weeks of real traffic, and move to provisioned mode once the traffic shape is understood and the savings justify the extra planning. The reverse also happens: a provisioned stream that needs frequent resharding for spiky traffic, or that was throttled during an unexpected peak, moves to on demand. On demand has its own limits, including a default ceiling on write throughput per stream that can be raised by request, so very large streams still need a conversation with AWS before a launch.

Hot shards

Because records are placed by the hash of their partition key, a stream's throughput depends on keys being spread evenly. If one key dominates, such as a single very active customer or a constant like the string default, all its records go to one shard, which hits its 1 MB per second limit while the other shards sit idle. Producers then receive ProvisionedThroughputExceededException errors and must retry, adding latency, even though the stream as a whole has plenty of capacity.

One tenant sent 40 percent of all records under one key, so shard 3 hit its limit and throttled while seven shards sat mostly idle.

The fix depends on whether ordering per key matters. If not, a random or high cardinality partition key, such as a request ID, spreads records evenly. If ordering per entity matters, the key should be the finest entity that needs ordering, such as a device or session rather than a whole tenant. When one entity truly needs more than a shard's capacity, key salting appends a small random suffix to spread it across several shards, at the cost of ordering only within each suffix. CloudWatch shard level metrics, enabled per stream, show which shards are hot and which keys need attention.

Detecting a hot key before it causes throttling is easier than diagnosing it afterwards. Producers can log or sample partition key frequencies, and a simple report of the top keys by volume, run during load tests and periodically in production, shows whether one key dominates. Kinesis does not report per key metrics, only per shard, and one shard can host many keys, so the producer side view is the only way to see which key is responsible. Teams that run multi tenant streams often keep a list of large tenants whose keys are salted from the start, rather than waiting for each to grow into a problem.

Resharding

In provisioned mode, capacity changes by splitting a shard into two, each taking half of its hash range, or merging two adjacent shards into one. UpdateShardCount performs a uniform scaling operation that splits or merges as needed to reach a target count. Resharding takes time, and the parent shard remains readable until consumers finish reading it, after which they continue with the children. Consumers must read the parent to its end before reading the children to preserve ordering, which the Kinesis Client Library handles automatically and hand written consumers often get wrong.

After the split, new writes went to the two children, and the consumer finished the parent before reading them, so order held.

There are limits on how often and how far resharding can go: UpdateShardCount can at most double or halve the count per call and runs a limited number of times per day. Scaling ahead of known peaks, such as a sales event, is safer than reacting during the peak. Some teams automate provisioned resharding from CloudWatch metrics with a Lambda function, but on demand mode has made that less common, since it handles the scaling and removes the risk of a resharding operation in progress when traffic spikes.

Producers

Producers write with PutRecord for single records or PutRecords for batches of up to 500 records and 5 MB. Batching reduces request overhead dramatically, and the Kinesis Producer Library aggregates many small user records into larger Kinesis records and batches them, raising throughput per shard for small records, with consumers deaggregating on read. PutRecords can partially fail, returning per record errors while other records succeed, so producers must check each result and retry only the failed records, with backoff, rather than assuming success or retrying the whole batch and creating duplicates.

Because producers retry, consumers may see duplicate records with different sequence numbers. Kinesis, like SQS, provides at least once delivery, so consumers should include a unique record ID from the producer and process idempotently. Producers that must not lose data during a Kinesis outage or throttling need a local buffer, bounded in size, and a decision about what to do when it fills: block the application, drop the oldest data, or spill to disk. That decision depends on whether the data is critical, such as orders, or tolerant of loss, such as debug telemetry.

Record size matters too. Kinesis records are limited to 1 MB, and payloads such as images, documents or large JSON blobs should be stored in S3 with only a reference written to the stream, a pattern often called a claim check. Small records have the opposite problem: a shard's 1,000 records per second limit is reached long before its 1 MB per second limit when each record is a few hundred bytes. Aggregation, through the Kinesis Producer Library or by combining several events into one record in application code, lets a shard carry its full byte capacity and lowers cost per event.

Keys decide throughput and order

Choose a partition key with enough distinct values to spread load and only as coarse as ordering requires. Watch shard level metrics for hot shards, use on demand for unpredictable traffic and provisioned for steady volume, and handle partial failures in PutRecords by retrying only the failed records.