Distributed Systems

Animated explainers on distributed systems.

All Caching & Performance Databases Kafka & Streaming Distributed Systems System Design Networking & Load Balancing ML & Agentic AI

Distributed Systems

May 14, 2026

Quorum Writes: How Distributed Systems Trade Latency for Consistency

Why waiting for every replica is a mistake. Quorum writes use the W + R > N rule to balance latency, availability, and consistency in distributed databases.

Distributed Systems

May 10, 2026

Three Kubernetes Probes, Three Different Jobs

Startup, readiness, and liveness probes look similar and do completely different things. Mixing them up is how a flaky dependency turns into a fleet-wide restart loop.

Distributed Systems

Apr 21, 2026

Kubernetes Ingress: One Door, Many Rooms, And the Controller That Actually Opens It

Ingress is the HTTP router at the cluster edge. The resource is just YAML. The Ingress Controller is what does the work. Get the class label wrong and nothing picks it up.

Distributed Systems

Apr 20, 2026

How Kubernetes Service Discovery Hides a Lot of Moving Parts Behind One DNS Name

A pod talks to payments.default.svc.cluster.local and it just works. Here is the CoreDNS, Service, and Endpoints machinery hiding behind that one stable name.

Distributed Systems

Apr 17, 2026

Three Replicas, One Node, Zero Availability: Why HA in Kubernetes Is About Placement

Replica count alone does not buy you HA. Where the replicas land does. Anti-affinity, topology spread, and PodDisruptionBudgets are the controls that turn three pods into actual redundancy.

Distributed Systems

Apr 15, 2026

What Actually Happens When a Kubernetes Node Dies

Automatic recovery sounds fast. In reality, a node failure costs you seconds-to-minutes of unavailability per pod, and the default timers explain why.

Distributed Systems

Apr 14, 2026

The Kubernetes Service Is the Whole Reason Pods Can Be Cattle

The Service object looks like a small piece of YAML, but it is the abstraction that lets pods churn freely without breaking every caller. Here is what it actually hides.

Distributed Systems

Apr 10, 2026

The Rolling Deploy That Returns 5xx: Why Readiness Probes Decide When Traffic Moves

A rolling update is not really about creating and deleting pods. It is about deciding when traffic shifts. Readiness probes are the gate, and getting them wrong burns users every release.

Distributed Systems

Apr 9, 2026

Pods Are Cattle: How Kubernetes Self-Healing Actually Works

Kubernetes does not restart dead pods. It creates new ones to satisfy a desired replica count. That distinction is the whole reason StatefulSets exist.

Distributed Systems

Apr 7, 2026

Circuit Breaker Pattern Deep Dive: The Half-Open State Is Where the Bugs Live

A circuit breaker does not heal a failing dependency, it contains the blast radius. The closed, open, and half-open states each have their own failure modes worth knowing.

Distributed Systems

Apr 4, 2026

Why Retries Need Idempotency: The Real Fix for Duplicate Payments and Orders

Retries recover from failures, but they can also duplicate them. Idempotency keys are what turn a dangerous retry into a safe one across payments and orders.

Distributed Systems

Mar 11, 2026

IP Routing and Longest Prefix Match: Why the Most Specific Route Always Wins

Routing tables are tries, not hash maps. The longest matching prefix wins because specificity equals authority, and BGP propagates that authority across the internet.

Distributed Systems

Mar 11, 2026

ARP and the L2/L3 Boundary: Why Your VIP Failover Sometimes Takes Four Minutes

ARP bridges IP addresses to MAC addresses on the local subnet. Gratuitous ARP is how failover tells the LAN about a new owner, except when a switch decides to ignore it.

Distributed Systems

Mar 5, 2026

Split Brain and Fencing Tokens: What Happens After the Election

Electing a new leader is the easy part. Preventing the old leader from writing once it wakes up from a long pause is what fencing tokens are for.

Distributed Systems

Mar 5, 2026

Raft Consensus: Leader Election, Log Replication, and a Slow-Disk Failure

Raft breaks consensus into leader election, log replication, and safety. Why majority quorum is magic, and how a slow follower took down a 3-node cluster.

Distributed Systems

Mar 4, 2026

Saga Pattern for Distributed Transactions: Compensations Need Idempotency Keys Too

Two-phase commit collapses under coordinator failures. Sagas replace global atomicity with compensating actions, but only if every forward step and every compensation is idempotent.

Distributed Systems

Mar 2, 2026

Causal Consistency vs Linearizability: Choosing the Right Order

Linearizability gives one global timeline at consensus cost. Causal consistency preserves happened-before without coordination. The tradeoff and one production trap.

Distributed Systems

Mar 1, 2026

Vector Clocks: Detecting Concurrent Writes Without a Global Clock

Vector clocks track happened-before across multi-writer replicas and flag true concurrency. Plus the sibling explosion that broke a key-value store in production.

Distributed Systems

Feb 27, 2026

Why Clocks Lie in Distributed Systems

NTP jumps, VM drift, and leap seconds make wall-clock ordering unsafe. Why Spanner waits out clock uncertainty, and how an 80 ms rewind erased 1200 financial orders.

Distributed Systems

Feb 27, 2026

Infra Cron vs Application Scheduler: Who Should Own the Job Lifecycle

Kubernetes CronJob and EventBridge are not interchangeable with Celery beat or TaskIQ. Picking wrong means either fighting your platform or losing visibility into your own jobs.

Distributed Systems

Feb 25, 2026

Read-Your-Writes Consistency: Why Your Profile Update Disappears on Refresh

You save a profile change, refresh, and see the old value. The bug is not the form. It is replication lag plus a follower read. Here are the four real fixes.

Distributed Systems

Feb 23, 2026

Kubernetes Is a Control Loop, Not a Container Runner

Kubernetes is a declarative reconciler. Every feature you think of as separate (probes, restarts, PDBs, spread) is a different specific control loop on top of the same mental model.

Distributed Systems

Feb 17, 2026

Consistent Hashing: Why Virtual Nodes Are Not Optional

Plain modulo hashing remaps almost every key when the cluster size changes. A hash ring moves only K/N keys, but without virtual nodes the load distribution is wildly uneven and gets worse when you add capacity.

Distributed Systems

Dec 27, 2025

Idempotency and Retries: The Pattern Every State-Changing API Owes You

Retries are a feature, but without idempotency they are a bug. Client-supplied idempotency keys plus a dedup table with TTL is the pattern Stripe and every payments API converged on for a reason.

Distributed Systems

Dec 26, 2025

Sagas: Distributed Transactions Without the Global Lock

Two-phase commit blocks. Three-phase commit assumes a network you do not have. Sagas accept eventual consistency and replace rollback with explicit compensating actions.

Distributed Systems

Dec 24, 2025

Three-Phase Commit: Why It Exists and Why Nobody Uses It

3PC was invented to fix the blocking problem in 2PC by adding a PreCommit phase. It works in theory and fails the moment you allow network partitions.

Distributed Systems

Dec 23, 2025

Three-Phase Commit vs Two-Phase Commit: Why Nobody Actually Runs 3PC

3PC adds a PreCommit phase to fix the blocking problem in 2PC, but the proof assumes a synchronous network. Real networks reorder and delay, and the protocol breaks in subtle ways.

Distributed Systems

Dec 22, 2025

Two-Phase Commit Walkthrough: The Blocking Problem Nobody Wants to Own

2PC gives you atomicity across services at the cost of availability. Participants that voted yes will hold their locks forever if the coordinator dies at the wrong moment.