Distributed Systems Basic

When an in-process function call fails, you know for sure it failed. When a network call times out, you know nothing: the request may not have left the client, may have reached the server but not run, may have committed, or may have committed with the response lost.

That is partial failure, and almost every hard topic in distributed systems stems from it.

Article map

Problem	Article	Key conclusion
A timeout does not tell you what the server did	Eight fallacies	An RPC has four failure points; operations with side effects need an idempotency key
Retrying without making the failure worse	Retry, backoff and jitter	Full jitter, retry budget, retry at only one layer
Machine clocks do not agree	Logical clocks	L(a) < L(b) does not prove causality; wall-clock LWW loses data
One slow dependency takes down the whole API	Circuit breaker & bulkhead	Timeouts are always needed; bulkheads isolate resources; a breaker fails fast when errors or slow calls cross a threshold
One business operation spans several services and a later step fails	Saga	Compensations are semantic undo, run in reverse order; persist saga state; steps and compensations must be idempotent

Suggested reading order: the eight fallacies, then retry (a direct consequence of partial failure), then logical clocks.

The same boundary in other sections

Section	How partial failure shows up
Kafka is a distributed log	Committing an offset does not confirm that a side effect outside Kafka succeeded
ACID in PostgreSQL	The connection drops right as COMMIT is sent: the client does not know whether the transaction committed
Cache-aside	Cache and database are two systems and are not updated atomically together
Timeout budget	A time budget for the whole retry chain

This section is still being expanded.