Kafka & Streaming

Animated explainers on kafka & streaming.

All Caching & Performance Databases Kafka & Streaming Distributed Systems System Design Networking & Load Balancing ML & Agentic AI

Kafka & Streaming

May 7, 2026

Kafka Consumer Group Rebalancing: How Partitions Get Reassigned

Kafka consumer groups stay running even when one consumer crashes. Here is how the group coordinator rebalances partitions and keeps the stream flowing.

Kafka & Streaming

May 6, 2026

How Kafka Replicates Data and Elects a New Leader When a Broker Dies

Kafka survives broker failures through partition replication and ISR-based leader election. Here is how the High Watermark and the controller keep data safe.

Kafka & Streaming

May 6, 2026

Kafka In-Sync Replicas Deep Dive: When ISR Shrinks and Writes Quietly Stop

The ISR is not a static list. It expands and contracts based on replica.lag.time.max.ms, and when it falls below min.insync.replicas your producers stop writing.

Kafka & Streaming

May 5, 2026

Kafka Exactly-Once Semantics: The Three Pieces You Actually Need

Kafka EOS is not one feature. It is the idempotent producer, transactions, and `read_committed` consumers working together across the full consume-process-produce loop.

Kafka & Streaming

May 3, 2026

CQRS with Async Projection: Two Models, One Truth

CQRS splits the write model from the read models and lets each be optimized in isolation. Async projections introduce lag, multiple read shapes, and a new operational concern: projection drift.

Kafka & Streaming

May 2, 2026

MapReduce Explained: Why a 20-Year-Old Idea Still Runs Your Batch Jobs

MapReduce is map, shuffle, reduce. Three phases, two functions, and a scheduler that makes embarrassingly parallel work cheap. Here is why Spark replaced it and where it still wins.

Kafka & Streaming

May 1, 2026

Tumbling vs Sliding vs Session Windows: Picking the Right Shape for Stream Aggregates

Tumbling windows give you clean billing buckets. Sliding windows give you smooth anomaly detection. Session windows give you user behavior. Pick by what question you are asking.

Kafka & Streaming

Apr 30, 2026

Exactly-Once Stream Processing: How Checkpoints and Idempotency Work Together

Exactly-once stream processing is not one feature. It is checkpoints plus idempotent sinks. Here is why you need both, and what breaks if you skip either.

Kafka & Streaming

Apr 5, 2026

Kafka Consumer Lag: When Your System Is Up but No Longer Real Time

Consumer lag is the gap between the log-end offset and the committed offset. It is a per-partition metric, and the worst kind of incident: not down, just slow.

Kafka & Streaming

Apr 5, 2026

Backpressure: How Healthy Systems Push Back Before Their Queues Eat Them

Systems rarely die from being busy. They die from accepting work they cannot finish. Backpressure is the mechanism that makes overload visible early.

Kafka & Streaming

Mar 9, 2026

Finding the Top-K Heavy Hitters in an Unbounded Stream

Count-Min Sketch plus a min-heap solves Top-K for unbounded streams. Here is how the approximation works and where the overestimates actually bite you.

Kafka & Streaming

Mar 2, 2026

At-Least-Once vs Exactly-Once: There Is No Third Option

Three delivery semantics exist on paper, but exactly-once is really at-least-once plus idempotency. Pick the one your business logic can survive.

Kafka & Streaming

Feb 27, 2026

DLQ and Retry Backoff: The Pattern That Saves Your Queue at 2am

A single poison message can stall a partition and stop every other consumer behind it. Tiered retries plus a DLQ keep the rest of the stream flowing.

Kafka & Streaming

Feb 25, 2026

Exactly-Once Delivery Is a Myth. Aim for Exactly-Once Effects Instead.

Exactly-once network delivery is impossible. What you actually want is exactly-once effects: at-least-once transport plus idempotent consumers with a dedupe key.

Kafka & Streaming

Jan 18, 2026

Redis Streams Trimming: The Tilde in MAXLEN Is Not Free

XADD MAXLEN ~ N is approximate and can keep nearly twice as many entries as you asked for. Under bursty writes that lie becomes silent data loss. Use exact bounds or move streams to their own instance.

Kafka & Streaming

Jan 17, 2026

Redis Streams Make Backpressure Visible, and That Is the Whole Point

The Pending Entries List is the load-bearing data structure. A slow consumer does not silently lose work, it accumulates entries you can measure with XPENDING and reassign with XAUTOCLAIM.

Kafka & Streaming

Jan 16, 2026

Redis Streams Are At-Least-Once. You Build Exactly-Once Effects on Top.

Redis Streams give you durable delivery and a pending entries list. They do not give you exactly-once. The trick is pairing them with idempotent sinks so duplicates do not cause damage.

Kafka & Streaming

Jan 16, 2026

Redis Pub/Sub vs Streams: Two Tools That Look Similar and Solve Opposite Problems

Pub/Sub is fire and forget with no memory. Streams are a durable log with consumer groups and replay. Picking the wrong one quietly loses messages until the day it does not.

Kafka & Streaming

Jan 6, 2026

Redis Is Three Messaging Systems Pretending to Be One

Pub/Sub is at-most-once and forgets offline subscribers. LIST plus BLPOP is at-least-once with manual ACK. Streams add consumer groups and replay. Picking wrong costs you messages.

Kafka & Streaming

Jan 4, 2026

Kafka Producer Performance Tuning: The Four Knobs That Actually Matter

Most Kafka producer throughput problems are a config problem, not a Kafka problem. Here are the four settings that turn a 50K msg/s producer into a 400K msg/s producer.

Kafka & Streaming

Jan 3, 2026

Kafka Partitioning and Ordering: Picking the Right Key

Kafka partitioning is three things at once: parallelism, ordering, and hot keys. The wrong key makes one consumer carry the whole load.

Kafka & Streaming

Jan 2, 2026

Kafka Consumer Offsets Deep Dive: Position vs Committed

Kafka consumers track two offsets per partition: an in-memory position and a durable committed offset. Auto-commit hides a five-second window where crashes cause duplicate processing.

Kafka & Streaming

Dec 31, 2025

Kafka Idempotent Producer: What `enable.idempotence=true` Actually Buys You

The idempotent producer dedupes retries by (producer_id, sequence_number) per partition. It is not exactly-once across services, and confusing the two causes real outages.

Kafka & Streaming

Dec 31, 2025

How Kafka Transactions Work: Atomicity by Visibility, Not Rollback

Kafka transactions never roll back. The coordinator writes COMMIT or ABORT markers and read_committed consumers skip the rest. Here is how that adds up to exactly-once.

Kafka & Streaming

Dec 30, 2025

Kafka Log Truncation and Unclean Leader Election: When Failover Rewrites History

Failover in Kafka is a log truncation event. Clean election drops only uncommitted data. Unclean election can erase records consumers already read.

Kafka & Streaming

Dec 28, 2025

Kafka Log Compaction vs Time Retention: Two Storage Models in One Broker

Time/size retention is for event streams. Compaction keeps the latest value per key for state topics. Pick the wrong one and your consumers drift silently.

Kafka & Streaming

Dec 27, 2025

The Transactional Outbox: Publishing Events Without Losing Them

Writing to a database and publishing an event are two operations. The transactional outbox makes them one, so events survive crashes and the broker can be down without anyone noticing.

Kafka & Streaming

Dec 25, 2025

Kafka acks, min.insync.replicas, and Producer Retries: The Durability Dial

Kafka durability is a dial, not a switch. acks, min.insync.replicas, and producer retries together decide whether a write survives a broker failure or vanishes.

Kafka & Streaming

Dec 22, 2025

Kafka Durability and acks: Three Settings, Three Different Guarantees

Kafka's acks setting is the difference between fire-and-forget, leader-only, and full ISR durability. Pair it with min.insync.replicas or it does not mean what you think.

Kafka & Streaming

Dec 21, 2025

Kafka Scaling Math: Partitions, Consumers, and Why More Is Not Free

Kafka throughput is partitions times per-partition ceiling. Consumer parallelism caps at partition count. Doubling partitions costs you controller load, rebalance time, and broker boot time.