Patterns of Distributed Systems Link to heading
Based on: Catalog of Patterns of Distributed Systems by Unmesh Joshi (published on martinfowler.com, Nov 2023)
Table of Contents Link to heading
Group F — Network Communication & Request Handling Link to heading
Reliable data structures and leader election aren’t enough. Nodes must communicate efficiently and correctly over unreliable networks.
F1. Single-Socket Channel Link to heading
One TCP connection per peer pair to preserve message order.
Problem Link to heading
When a node sends multiple messages to another node using separate connections (or UDP), messages can arrive out of order. For many distributed protocols (like log replication), ordering is critical: entry 5 must be applied before entry 6.
Solution Link to heading
Maintain one persistent TCP connection between each pair of communicating nodes. TCP guarantees in-order delivery within a single connection, so messages always arrive in the order sent.
How it works Link to heading
Node A ◄═══ single TCP socket ═══► Node B
(reused for all messages between A and B)
Message ordering:
A sends: [msg1, msg2, msg3]
B receives: [msg1, msg2, msg3] ← guaranteed by TCP ✓
Connection lifecycle:
- Connection established at startup or on first message
- Reused indefinitely (with heartbeats to detect dead connections)
- On disconnect, reconnect and resume from the right position (using log indices)
Real-world analogy Link to heading
A dedicated phone line between two offices used for all communications, rather than dialing a new number for each call. Everything said on that line is heard in order.
Example systems Link to heading
ZooKeeper (single connection per client), Raft implementations, etcd
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Guaranteed message ordering at no extra cost | Head-of-line blocking: a slow message delays all subsequent messages |
| Simple — no sequence numbering needed | Connection management complexity (reconnection, multiplexing) |
| Efficient (reuse TCP handshake cost) | Single socket limits throughput if messages are large |
Related patterns Link to heading
→ Request Pipeline (send multiple requests without waiting for responses) → HeartBeat (keeps the socket alive) → Singular Update Queue (processes messages from the socket serially)
F2. Request Pipeline Link to heading
Don’t wait for acknowledgement before sending the next request.
Problem Link to heading
Using a Single-Socket Channel, if you send a request and wait for a response before sending the next request, your throughput is limited by the round-trip latency. If latency is 10ms and requests take 1ms to process, you can only do 100 requests/second even if the server could handle 10,000.
Solution Link to heading
Pipeline multiple requests: send them one after another without waiting for the previous response. The server processes them in order (FIFO) and returns responses in the same order.
How it works Link to heading
Without pipelining:
Send Req1 → wait → Recv Resp1 → Send Req2 → wait → Recv Resp2
Time = 2 × RTT per operation
With pipelining:
Send Req1 → Send Req2 → Send Req3 → ...
→ Recv Resp1 → Recv Resp2 → Recv Resp3
Time ≈ 1 × RTT for many operations
The sender keeps an in-flight window (e.g., up to 100 outstanding requests). If the window fills up, it pauses until acknowledgements arrive.
Real-world analogy Link to heading
An assembly line vs. building one product at a time. You start work on the next unit before the first is finished — the stages overlap.
Example systems Link to heading
Redis pipelining, HTTP/2 multiplexing, Raft leader-to-follower replication
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Dramatically improves throughput | Errors are harder to handle (which request failed?) |
| Reduces effective latency under load | Window sizing is tricky — too large can overwhelm receiver |
| Utilizes network bandwidth efficiently | Out-of-order responses (if using multiplexed channels) require correlation IDs |
Related patterns Link to heading
→ Single-Socket Channel (provides the ordered channel) → Request Batch (complementary throughput optimization) → Request Waiting List (tracks in-flight requests)
F3. Request Batch Link to heading
Bundle multiple small requests into one large request.
Problem Link to heading
Sending many small requests incurs high overhead: each request has network framing, serialization, and processing overhead. When throughput is critical (e.g., log replication), the overhead per message can dominate actual work time.
Solution Link to heading
Accumulate multiple pending requests and send them as a single batch. The receiver processes the batch together and sends a single response for the batch.
How it works Link to heading
Without batching (N small requests):
[req1] → network → [resp1]
[req2] → network → [resp2]
[req3] → network → [resp3]
Cost: N × (serialization + network overhead)
With batching:
[req1 + req2 + req3] → network → [resp1 + resp2 + resp3]
Cost: 1 × (serialization + network overhead) + 3 × processing
Batching strategies:
- Size-based: batch when accumulated size reaches N bytes
- Time-based: batch every T milliseconds (e.g., 5ms)
- Hybrid: batch when size OR time threshold is reached
Real-world analogy Link to heading
A courier service: instead of sending 50 individual packages across the city one by one, fill a van and deliver them all in one trip.
Example systems Link to heading
Kafka (producer batching), database write batching, Raft heartbeat piggybacking
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Major throughput improvements | Increases latency for individual requests (must wait for batch to fill) |
| Reduces per-request overhead | More complex error handling (partial batch failures) |
| Efficient network utilization | Buffering introduces memory overhead |
Related patterns Link to heading
→ Request Pipeline (send batches in a pipeline) → Singular Update Queue (where batches are assembled) → Write-Ahead Log (batch writes to the log)
F4. Singular Update Queue Link to heading
Process state changes sequentially using a single worker thread.
Problem Link to heading
Multiple threads or network handlers updating shared state concurrently require locks, condition variables, and careful synchronization — a recipe for deadlocks, race conditions, and hard-to-test code.
Solution Link to heading
Route all state-changing operations through a single queue processed by one dedicated thread. This thread processes one request at a time, eliminating all concurrency issues around shared state. Callers submit tasks to the queue and optionally wait for a result via a future/promise.
How it works Link to heading
Thread 1 ─────► ┌─────────────────────┐
Thread 2 ─────► │ Request Queue │ ◄──── Single Worker Thread
Thread 3 ─────► │ [req1, req2, req3] │ (processes serially)
└─────────────────────┘
Network I/O and request parsing are still multi-threaded. Only state mutation is serialized through the queue.
This is the actor model in miniature.
Real-world analogy Link to heading
A restaurant kitchen has one expediter who calls out orders one at a time to the kitchen staff. No two orders are processed simultaneously — each is completed before the next is started. Waiters (multiple threads) can still take orders concurrently.
Example systems Link to heading
etcd (single Raft goroutine), ZooKeeper, Actor frameworks (Akka), Node.js event loop
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Eliminates concurrency bugs on shared state | Single thread is a throughput ceiling |
| Dramatically simplified code — no locks needed | Queue can become a bottleneck under high load |
| Easy to reason about state transitions | Slow operations in the queue block everything |
Related patterns Link to heading
→ Request Waiting List (tracks async responses from the queue) → Request Batch (queue can batch work) → Single-Socket Channel (feeds the queue)
F5. Request Waiting List Link to heading
Track in-flight requests waiting for responses from the cluster.
Problem Link to heading
A server receives a client request and must wait for acknowledgements from multiple cluster nodes before it can respond. The server is asynchronous — it doesn’t block waiting — so it must remember which client is waiting for which cluster responses.
Solution Link to heading
Maintain a waiting list (a map) of in-flight client requests. Each entry stores:
- The original client request
- A callback or future to invoke when the required conditions are met (e.g., majority acks)
- A timeout
When cluster node responses arrive, match them to waiting list entries, check if the success criterion is met, and either respond to the client or trigger retry logic.
How it works Link to heading
Client sends write request → Server creates WaitEntry {
requestId: "abc123",
requiredAcks: 3, (majority of 5)
receivedAcks: [],
timeout: 5000ms,
clientCallback: fn
}
Server stores WaitEntry in map["abc123"]
Follower1 acks → map["abc123"].receivedAcks.push(1) → count=1
Follower2 acks → map["abc123"].receivedAcks.push(2) → count=2
Follower3 acks → map["abc123"].receivedAcks.push(3) → count=3 = majority
→ clientCallback("SUCCESS") → respond to client
→ remove from waiting list
Real-world analogy Link to heading
A restaurant manager tracks each table’s order. When the kitchen finishes each dish, the manager checks if the whole table’s order is complete. Only when all dishes are ready does the server bring them out.
Example systems Link to heading
Raft leader implementations, ZooKeeper request processors, custom quorum write handlers
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Non-blocking — server doesn’t stall | Must handle timeouts explicitly |
| Clean separation of request receipt and response | Memory grows with in-flight requests |
| Natural place to implement retry and timeout logic | Correlating responses with requests requires unique IDs |
Related patterns Link to heading
→ Singular Update Queue (submits tasks that eventually respond via the waiting list) → Request Pipeline (multiple in-flight requests simultaneously) → Idempotent Receiver (handle retries when waiting list times out)
F6. Idempotent Receiver Link to heading
Make servers safe to retry: process the same request twice = process it once.
Problem Link to heading
Networks are unreliable. A client sends a request, the server processes it, but the response is lost. The client times out and retries. The server processes the request again — potentially double-charging a credit card, double-sending an email, or creating duplicate records.
Solution Link to heading
Assign each request a unique client ID + request ID pair. The server tracks which requests it has already processed (in a deduplication store, usually persisted). On receiving a request, check if its ID has been seen before:
- If yes → return the cached response (don’t re-execute)
- If no → process and store the result
How it works Link to heading
Client:
requestId = UUID() ← unique per request attempt
Send: {clientId: "client-1", requestId: "req-abc", data: "charge $100"}
Server:
Check dedup_store["client-1"]["req-abc"]
→ Not found → process → charge $100 → response = "OK, charge ID=XYZ"
→ Store dedup_store["client-1"]["req-abc"] = "OK, charge ID=XYZ"
→ Return "OK, charge ID=XYZ"
Client retry (response was lost):
Send again: {clientId: "client-1", requestId: "req-abc", data: "charge $100"}
Server:
→ Found in dedup_store → return "OK, charge ID=XYZ" (no re-execution)
The dedup store has a TTL — old entries can be cleaned up once the client is done retrying.
Real-world analogy Link to heading
A bank check with a unique check number. If you lose the confirmation that your check was cashed and submit it again, the bank recognizes the duplicate check number and rejects the second submission.
Example systems Link to heading
AWS API Gateway (idempotency keys), Stripe (idempotency keys), Kafka exactly-once semantics, database UPSERT with unique constraints
Trade-offs Link to heading
| Pro | Con |
|---|---|
| Safe retries — clients can always retry on timeout | Requires persistent deduplication storage |
| Simplifies client logic (just retry on any failure) | Dedup store must be consulted on every request (overhead) |
| Enables exactly-once semantics | Old dedup entries must be cleaned up eventually |
Related patterns Link to heading
→ Request Waiting List (server retries internally; idempotency handles external retries) → Write-Ahead Log (dedup results are persisted here) → Generation Clock (can be used to detect outdated requests)