Patterns of Distributed Systems Link to heading

Based on: Catalog of Patterns of Distributed Systems by Unmesh Joshi (published on martinfowler.com, Nov 2023)

Table of Contents Link to heading

1. Group G — Data Propagation & Observation
   • Gossip Dissemination
   • Follower Reads
   • State Watch

Group G — Data Propagation & Observation Link to heading

These patterns handle how information spreads through a cluster and how clients track cluster state changes.

G1. Gossip Dissemination Link to heading

Spread information through the cluster like an epidemic — without a central broker.

Problem Link to heading

In a large cluster (hundreds or thousands of nodes), you need to disseminate information to all nodes (e.g., “Node X has failed,” “the new cluster configuration”). Broadcasting to every node is O(N) messages from a single source — it floods the network and creates a bottleneck.

Solution Link to heading

Each node periodically picks a random subset of peers and exchanges state with them. Each peer then spreads the information to other random peers. Information propagates exponentially fast (like a rumor spreading through a crowd) with no central coordinator.

How it works Link to heading

Time 0: Node A knows new info
Time 1: A gossips to [B, E] → B and E learn it
Time 2: B gossips to [C, F], E gossips to [D, G] → 6 nodes know
Time 3: C, D, F, G gossip to others → 14 nodes know
...
After O(log N) rounds: all N nodes know

Each gossip message includes a version number or digest so nodes can tell what information is new vs. already known. This prevents infinite re-gossip.

Real-world analogy Link to heading

Office rumors. Person A tells two colleagues. Each of them tells two more. Within a few hours, the whole building knows — without anyone sending a company-wide broadcast.

Example systems Link to heading

Amazon Dynamo (membership and failure detection), Cassandra (cluster metadata), Consul, Riak, Bitcoin peer discovery

Trade-offs Link to heading

Related patterns Link to heading

→ HeartBeat (often piggybacked in gossip messages) → Version Vector / Versioned Value (for deduplication in gossip) → Emergent Leader (uses gossip for membership info) → Consistent Core (alternative to gossip for strongly consistent metadata)

G2. Follower Reads Link to heading

Scale read throughput by serving reads from follower nodes.

Problem Link to heading

In a Leader and Followers setup, all writes and strong reads go to the leader. As read traffic grows, the leader becomes a bottleneck. You have multiple follower nodes sitting idle for reads.

Solution Link to heading

Allow clients to read from follower nodes. Followers may be slightly behind the leader (stale data), but for many use cases, slightly stale reads are acceptable — and the throughput gain is significant.

How it works Link to heading

Write path:  Client → Leader → replicate to followers
Read path:
  - Strong read:   Client → Leader (always up-to-date)
  - Follower read: Client → any Follower (may be slightly stale)

Client can specify: read-my-own-writes?
  If yes: read from leader, or wait for follower to catch up to your last write's index
  If no: any follower is fine

Read-your-own-writes consistency: After a write, the client can pass the log index of their write to the follower. The follower waits until it has applied entries up to that index before responding.

Real-world analogy Link to heading

A company’s website: the main database handles all updates. Reporting dashboards and analytics pages can read from a replica that’s a few seconds behind — nobody cares if the “total users” count is 3 seconds stale.

Example systems Link to heading

MySQL read replicas, MongoDB secondary reads, Cassandra (tunable consistency), CockroachDB follower reads

Trade-offs Link to heading

Related patterns Link to heading

→ Leader and Followers → High-Water Mark (follower knows how far behind it is) → Versioned Value (client can request specific version for consistency)

G3. State Watch Link to heading

Push notifications: tell me when something changes.

Problem Link to heading

Clients often need to react to changes in cluster state (e.g., “tell me when the leader changes,” “tell me when a configuration key is updated”). Polling — repeatedly asking “has it changed?” — is wasteful: it generates load even when nothing has changed and adds latency to detect changes.

Solution Link to heading

Clients register a watch (subscription) on a specific key or path in the consistent store. When that key’s value changes, the server pushes a notification to all registered watchers. No polling required.

How it works Link to heading

Client A:  "Watch key /config/leader"
  → Server registers watch: {key: "/config/leader", callback: A.notify}

[Some time later...]

New leader elected → Server writes "/config/leader" = "Node5"

Server iterates watches for "/config/leader":
  → Sends notification to Client A: {key: "/config/leader", newValue: "Node5"}

Client A receives notification → updates its cached leader address
  → Client A can re-register the watch if it wants future notifications

Watches are typically one-shot: they fire once and must be re-registered. This prevents memory leaks from forgotten watches.

Real-world analogy Link to heading

A stock price alert: you set an alert at “$AAPL crosses $200.” Your broker notifies you once when it crosses — you don’t have to check the price every second.

Example systems Link to heading

ZooKeeper (watches), etcd (watch API), Kubernetes controller-manager (watches the API server), Consul watches

Trade-offs Link to heading

Related patterns Link to heading

→ Consistent Core (provides the watch service) → Lease (watches combined with leases implement leader election in ZooKeeper)