Fact-checked revision dated 27 October 2026, based on the original article published 15 September 2026. First-person wording and the Alex/Coding with Alex byline belong to the original author, not the translator or editor.
Hey everyone, Alex here. Welcome back to Coding with Alex!
A reading list of distributed-systems classics can span Lamport’s logical clocks, Paxos, Raft and eventual-consistency models. These foundational papers are worth revisiting even when the infrastructure is managed for you.
At first glance, it is easy for developers to dismiss these papers as academic ancient history. We live in the golden era of cloud infrastructure, don't we? If we need a highly available, globally distributed database, we spin up AWS DynamoDB or Google Cloud Spanner. If we need a message broker, we provision managed Kafka. We’ve abstracted away the metal, the network, and the consensus algorithms behind slick SDKs and Terraform scripts.
But here is the hard truth: the abstractions we rely on are leaky. If a network partition occurs or a node silently drops offline, those slick SDKs start throwing cryptic errors. If you don't understand the classical principles governing these systems, you will find yourself staring at production logs at 3:00 AM, desperately trying to figure out why your microservices are in a split-brain state.
Correction sources: [13]
Today, we are going to dive into why these distributed systems classics are more relevant now than ever. We'll look at three core concepts from these papers—Logical Time, Consensus, and the CAP/PACELC trade-offs—and map them directly to the real-world bugs you encounter in modern web development and cloud engineering.
The Fallacy of Wall-Clock Time: Why System.currentTimeMillis() Will Betray You
Let’s start with a classic scenario. Imagine you are building a collaborative document editing tool or a high-throughput financial ledger. Two API requests hit two different server instances in your cluster to update the same record. To determine which update happened first, you simply capture the system timestamp of each request and compare them. Easy, right?
This is where Leslie Lamport’s seminal 1978 paper, "Time, Clocks, and the Ordering of Events in a Distributed System", enters the room to shatter your dreams.
In a distributed system, do not assume that physical wall-clock timestamps provide a reliable global ordering of events. Each server has a local clock whose oscillator can drift; temperature and hardware characteristics affect that drift, while variable network delay adds uncertainty to synchronization. Even if you use Network Time Protocol (NTP) to synchronize your servers, NTP itself is subject to network delays and can cause "clock steps" (where the system clock is suddenly jumped forward or backward to sync with a time server).
If Server A’s clock is drift-skewed 50 milliseconds behind Server B’s, an event that occurred on Server A after an event on Server B might be assigned an earlier timestamp. In a last-write-wins database, this can cause a later real-world update to lose to an earlier one.
Correction sources: [1]
The Solution: Lamport Clocks
Lamport clocks assign logical timestamps that respect the happens-before relation: if a happens before b, then C(a) < C(b). The converse does not follow, so scalar Lamport timestamps alone cannot detect concurrency. A Lamport logical clock is a simple, monotonically increasing counter maintained by each process.
Correction sources: [12]
Here are the rules of a Lamport Clock:
- Each process increments its counter before executing an internal event.
- When process P₁ sends a message to P₂, it includes its current clock value C.
- When P₂ receives the message, it updates its clock to max(local_clock, C) + 1.
Let's look at a simple Python implementation of a Lamport Clock to see how this works in practice:
class LamportClock:
def __init__(self, process_id):
self.value = 0
self.process_id = process_id
def increment(self):
self.value += 1
return self.value
def send_event(self):
self.increment()
print(f"[Process {self.process_id}] Sent message. Clock: {self.value}")
return self.value
def receive_event(self, received_timestamp):
self.value = max(self.value, received_timestamp) + 1
print(f"[Process {self.process_id}] Received message. Clock updated to: {self.value}")
return self.value
Vector clocks and dotted version vectors can track causal histories and help identify concurrent versions, as documented for Riak KV 2.2. Cassandra instead uses client- or coordinator-supplied timestamps and last-write-wins conflict resolution, so clock synchronization matters. Next time you configure a database cluster, remember Lamport: order is about causality, not clocks.
Consensus in Action: Why We Can't Just "Ping" to Check if a Node is Dead
A possible cause of infrastructure outages is the "split-brain" scenario. Imagine a three-node database cluster (Node A, Node B, Node C) storing user sessions. Suddenly, a network switch fails, isolating Node A from Nodes B and C.
Correction sources: [13]
Node A can't talk to B or C. It wonders: "Are B and C dead, or am I isolated?"
Nodes B and C can talk to each other, but not to A. They wonder: "Is A dead, or are we isolated?"
If Node A assumes B and C are dead, it might try to elect itself as the leader and continue accepting write requests. Meanwhile, Nodes B and C elect B as the leader and also accept writes. When the network heals, you have two conflicting datasets that cannot be easily merged. Your database has split its brain.
The Raft Protocol: Consensus Made Human-Readable
To solve this, distributed systems rely on consensus protocols. While Paxos is the legendary, notoriously difficult-to-understand classic paper on this topic, Diego Ongaro and John Ousterhout introduced Raft in 2014 as an understandable consensus algorithm designed for real-world implementations.
Raft uses majority votes to elect a leader and majority replication, subject to its log-commitment rules, to commit entries. A majority quorum is Q = ⌊N/2⌋ + 1 voting members.
Correction sources: [13]
In our three-node cluster, a quorum is ⌊3/2⌋ + 1 = 2 nodes.
[Isolated Partition] [Active Partition] +------------------------+ +--------------------------+ | Node A | | Node B Node C | | (No Quorum: 1/3 nodes) | X | (Has Quorum: 2/3 nodes) | | - Cannot accept writes| | - Elects Leader | | - Read-only or offline| | - Safe to accept writes | +------------------------+ +--------------------------+
Node A cannot commit new writes on its own because it lacks a majority, even if it has not yet recognized that it is no longer the effective leader. Nodes B and C can form a majority, elect a leader and commit writes. In the diagram, “accept writes” means successfully commit them, not merely receive a client request.
Correction sources: [13]
When you configure Consul or etcd, you are working with systems that use Raft. ZooKeeper uses the related but distinct Zab protocol. For majority voting, an odd number of voting members—typically 3, 5, or 7—is usually the efficient choice, not a requirement. At least N = 2F + 1 voting members are needed to tolerate F unavailable members while retaining a majority.
Beyond CAP: The PACELC Theorem
CAP states that during a network partition, a system cannot guarantee both linearizable consistency and availability for every request to a non-failing node. Linearizability requires operations to appear to occur in a single order consistent with real-time precedence; availability requires those requests eventually to complete.
Correction sources: [9]
CAP describes a limit during partitions; it does not by itself describe all the trade-offs during normal operation. What happens when the network is communicating normally?
Correction sources: [9]
Daniel Abadi introduced the PACELC formulation in a 2010 blog post and developed it in a 2012 paper. It broadens the discussion to latency and consistency during normal operation:
If there is a Partition (P), how does your system trade off Availability (A) and Consistency (C); Else (E), how does your system trade off Latency (L) and Consistency (C)?
Consider MongoDB: linearizable single-document reads combined with majority-acknowledged writes provide real-time ordering under the documented conditions. This requires coordination, but not acknowledgment from every replica. The latency cost depends on topology and the selected read and write concerns.
Correction sources: [6]
A configuration that acknowledges a write before every replica is updated can reduce waiting, but reads from a lagging replica may return older data. This is one possible design; acknowledgment and consistency guarantees are specific to the database and operation.
Evaluating Real Databases via PACELC
When designing your database schema and selecting a cloud storage engine, you should explicitly map your requirements to the PACELC spectrum:
- PC/EC: Strongly consistent operating modes prioritize consistency during partitions and accept coordination costs during normal operation. Spanner and CockroachDB provide strong reads, but also expose stale-read options; assess the chosen operation and mode rather than treating the product name as a complete classification.
- PA/EL: A configuration may favor availability during partitions and low latency during normal operation. Do not assign this label to all DynamoDB or Cassandra operations: DynamoDB global tables offer MREC and MRSC modes, and Cassandra exposes configurable consistency levels.
Wrapping Up: Why You Should Read the Classics
As software engineers, it is easy to get caught up in the hype cycle of new frameworks, languages, and cloud offerings. But the underlying physics of networking, hardware limitations, and mathematics don't change. The challenges we face today in distributed cloud architectures are the exact same problems that Leslie Lamport, Barbara Liskov, and Jim Gray solved decades ago.
The next time you are debugging a flaky microservice integration, configuring a database replication lag alert, or deciding on a consistency level for your cloud storage, don't just guess. Look to the classics. The answers are already there, written in elegant, time-tested papers.
Have you ever run into a nasty clock-skew or split-brain bug in production? What classical distributed systems paper helped you understand it? Let’s chat in the comments below!
Until next time, keep coding,
Alex
Sources and fact checking
Sources checked on 27 October 2026. Factual inaccuracies were corrected; examples were not executed.
- Apache Cassandra: Dynamo / Data Versioning
- Riak KV 2.2 — Conflict resolution
- etcd FAQ
- HashiCorp — Consul consensus
- Apache ZooKeeper: Zab vs. Paxos
- MongoDB Read Concern
- Daniel Abadi: Replication and the latency-consistency tradeoff
- AWS DynamoDB global tables design
- Daniel Abadi — Consistency Tradeoffs (2012)
- Google Cloud — Spanner consistency
- Cockroach Labs — Follower reads
- Lamport: Time, Clocks and the Ordering of Events in a Distributed System
- Ongaro and Ousterhout: In Search of an Understandable Consensus Algorithm
- NTP Clock State Machine
- Daniel Abadi — PACELC introduction (2010)