Patterns of Distributed Systems Link to heading
Based on: Catalog of Patterns of Distributed Systems by Unmesh Joshi (published on martinfowler.com, Nov 2023)
Table of Contents Link to heading
Group B — Cluster Coordination & Leadership Link to heading
Once individual nodes can store state reliably, you need them to act as a team. These patterns manage cluster membership, detect failures, and elect a single coordinator.
B1. HeartBeat Link to heading
The simplest liveness detector.
Problem Link to heading
In a distributed cluster, nodes need to know which other nodes are still alive and reachable. There is no reliable way to distinguish between “that node is very slow” and “that node has crashed.”
Solution Link to heading
Each server periodically sends a small heartbeat message to every other server (or to a central coordinator). If a node does not receive a heartbeat from a peer within a configured timeout, it marks that peer as unavailable.
How it works Link to heading
Node A ──── heartbeat (every 1s) ────► Node B
Node A ◄─── heartbeat (every 1s) ──── Node B
If Node A doesn't hear from Node B for > 3s:
Node A marks Node B as SUSPECTED DEAD
(may trigger leader election)
Configuration knobs:
- Interval: How often to send (e.g., every 500ms)
- Timeout: How long to wait before declaring failure (e.g., 3× interval)
Real-world analogy Link to heading
A fleet of ships uses radio check-ins every hour. If a ship misses three consecutive check-ins, it is declared in distress.
Example systems Link to heading
ZooKeeper sessions, Kubernetes node liveness probes, Raft, HDFS DataNode heartbeats
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Simple to implement | Network congestion can cause false positives |
| Works under all failure conditions | Timeout tuning is tricky — too short → false alarms; too long → slow failure detection |
| No external dependencies | Generates constant background traffic |
Related patterns Link to heading
→ Leader and Followers (heartbeats from followers to leader) → Lease (heartbeats renew leases) → Generation Clock (heartbeats carry generation numbers)
B2. Leader and Followers Link to heading
The workhorse of distributed coordination.
Problem Link to heading
When multiple nodes can accept writes simultaneously, you risk conflicts: two nodes might apply the same update in a different order, leading to divergent state.
Solution Link to heading
Designate exactly one node as the leader (also called primary, master). All writes go through the leader, which ensures a single total ordering of operations. Other nodes are followers (replicas, secondaries) that apply operations in the same order.
How it works Link to heading
┌─────────────┐
Client ──►│ LEADER │── replicate ──► Follower 1
│ (Primary) │── replicate ──► Follower 2
└─────────────┘── replicate ──► Follower 3
Write path:
- Client sends write to leader
- Leader appends to its log
- Leader replicates to followers (synchronously or asynchronously)
- Once quorum confirms, leader commits and responds to client
Read path:
- Strong reads → always from leader
- Stale reads → from followers (see Follower Reads)
Real-world analogy Link to heading
A committee has a chairperson. Proposals must go through the chair, who ensures all members process business in the same order.
Example systems Link to heading
MySQL primary-replica, MongoDB primary-secondary, Raft’s leader, ZooKeeper’s leader
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Simple to reason about (single ordering point) | Leader is a bottleneck for write throughput |
| Strong consistency easily achievable | Leader failure requires election (unavailability window) |
| Straightforward conflict resolution | All writes travel to the leader (latency for geographically distributed clients) |
Related patterns Link to heading
→ HeartBeat (detect leader failures) → Generation Clock (identify the current leader term) → Majority Quorum (how replication is confirmed) → Replicated Log (the mechanism for replication) → Follower Reads (scaling reads)
B3. Generation Clock Link to heading
A monotonic counter that tracks “who is the current leader.”
Problem Link to heading
After a leader crash and re-election, the old leader might come back online (e.g., after a network partition heals) and believe it is still the leader. This split-brain scenario leads to two nodes both accepting writes, causing data divergence.
Solution Link to heading
Maintain a generation number (also called epoch, term, or ballot number) that increments every time a new leader is elected. Every message is tagged with the sender’s generation number. Any node that receives a message with a higher generation than its own updates its state and defers to the new leader. Any message from a lower generation is rejected.
How it works Link to heading
Election 1: Leader A wins → generation = 1
[All messages tagged with generation=1]
Leader A crashes. Network partition heals.
Election 2: Leader B wins → generation = 2
[All messages tagged with generation=2]
Leader A comes back online with generation=1
Leader A sends a write (generation=1)
Followers see: current generation is 2 > 1 → REJECT A's message
Leader A steps down
Real-world analogy Link to heading
A government election cycle. A president from the previous term (term 44) cannot issue executive orders after a new president takes office (term 45). The term number on the order invalidates it.
Example systems Link to heading
Raft (term), Paxos (ballot number), Zookeeper ZAB (epoch), Kafka (controller epoch)
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Prevents split-brain elegantly | Generation number must be persisted to survive restarts |
| Enables safe leader changes | Requires coordination to agree on the new generation |
| Can be used to version any durable state | — |
Related patterns Link to heading
→ Leader and Followers → Write-Ahead Log (persists the generation) → Paxos / Raft (use generation/term extensively)
B4. Emergent Leader Link to heading
Problem Link to heading
Running a full leader election algorithm (like Paxos or Raft) is complex, slow, and requires a quorum. For some use cases, you need a simpler way to designate a coordinator, especially at cluster startup.
Solution Link to heading
Order all cluster nodes by their age (how long they have been a member of the cluster). The oldest living node automatically becomes the leader without any explicit voting. When the current leader dies, the next oldest takes over.
How it works Link to heading
Nodes join in this order:
Node A (joined at t=100) ← oldest → becomes leader
Node B (joined at t=200)
Node C (joined at t=300)
Node A dies → Node B is now oldest → becomes leader automatically
Nodes learn each other’s ages through cluster membership information propagated via Gossip Dissemination.
Real-world analogy Link to heading
In a traditional workplace without a named manager, the most senior employee by tenure naturally takes responsibility.
Example systems Link to heading
Akka Cluster (oldest node becomes singleton actor), some service mesh implementations
Trade-offs Link to heading
| Pro | Con |
|---|---|
| No explicit election protocol | Only works when cluster membership is known |
| Deterministic — no voting rounds | Leader changes are implicit, not announced |
| Works well in stable clusters | New nodes always start as followers, regardless of capability |
Related patterns Link to heading
→ Gossip Dissemination (propagate age/membership info) → Generation Clock (still needed to handle stale leaders)
B5. Lease Link to heading
A time-limited grant of authority.
Problem Link to heading
A leader needs to hold an exclusive lock on a resource (e.g., “I am the master and I alone will write to this shard”). But if the leader crashes, the lock must be automatically released so a new leader can take over. Traditional locks require the holder to explicitly release them — which is impossible if the holder crashes.
Solution Link to heading
A lease is a lock with a built-in expiration time (TTL). The lease holder must continuously renew the lease (usually via heartbeat) before it expires. If the holder crashes, the lease simply expires and another node can acquire it.
How it works Link to heading
Lease server grants lease to Node A with TTL = 10 seconds
Node A must send renewal every 5 seconds:
t=0 → A acquires lease (expires at t=10)
t=5 → A renews lease (expires at t=15)
t=10 → A renews lease (expires at t=20)
... A crashes at t=12 ...
t=20 → Lease expires → Node B can acquire it
Critical nuance: Node A must stop using the lease before it expires — not when it renews. This handles the case where A is alive but partitioned from the lease server.
Real-world analogy Link to heading
A library book loan. You can borrow it for 3 weeks and renew it. If you don’t return or renew it, someone else can borrow it automatically.
Example systems Link to heading
etcd leases (used by Kubernetes leader election), ZooKeeper ephemeral nodes, Chubby (Google’s lock service)
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Crash-safe locking without explicit release | Clock skew can cause subtle bugs |
| Self-healing (expired leases auto-release) | Brief unavailability window when lease expires |
| Used to fence off old leaders (fencing tokens) | Lease holder must be conservative about when to stop |
Related patterns Link to heading
→ HeartBeat (renews the lease) → Generation Clock (lease version acts as generation) → Consistent Core (often the service that grants leases)