Hey everyone, Alex here. Welcome back to another edition of Coding with Alex at sysseder.com.
Today, a fascinating historical deep-dive popped up on my Hacker News feed: "The Succession Crisis That Tore England Apart"—an exploration of "The Anarchy" of the 12th century, where a lack of a clear, agreed-upon successor to the throne plunged an entire nation into decades of civil war. As I was reading about Henry I, Empress Matilda, and the breakdown of royal authority, my developer brain couldn't help but draw a parallel.
What is a succession crisis in medieval England if not the ultimate real-world split-brain scenario?
In distributed systems, we face our own succession crises every single day. When a database primary node dies, how does the cluster decide who rules next? If two nodes both claim the "crown" of being the writable primary, we get split-brain, data corruption, and catastrophic downtime. In this post, we are going to look at how modern distributed databases solve the "succession crisis" using consensus algorithms like Raft and Paxos, write a custom leader election mechanism in Go, and talk about how to design your cloud infrastructure to survive the digital equivalent of a civil war.
The Anatomy of a Distributed Succession Crisis
In a simple active-passive database setup, you have a leader (the monarch) and several followers (the heirs). The leader handles all writes, replicates them to the followers, and maintains order. But what happens when the network partition hits?
Imagine a three-node cluster: Node A, Node B, and Node C. Node A is the leader. Suddenly, a network switch fails, isolating Node A from Nodes B and C.
- The isolated leader's perspective: Node A thinks, "I'm still alive, but I can't hear my followers. I must keep accepting writes."
- The isolated followers' perspective: Nodes B and C think, "The leader is dead. We must elect a new one."
If Nodes B and C elect Node B as the new leader, and both Node A and Node B are accepting writes, your system has split-brain. When the network heals, reconciliating those conflicting writes is an absolute nightmare that usually ends in data loss. To prevent this, we need a rigorous, mathematically proven framework for succession. Enter: Distributed Consensus.
The King is Dead, Long Live the King: How Raft Handles Succession
While Paxos is the grandfather of consensus, the Raft algorithm is what powers most modern developer tools like etcd (which runs Kubernetes), Consul, and CockroachDB. Raft is explicitly designed to be understandable, and its leader election phase is a masterclass in solving succession crises.
In Raft, a node can be in one of three states: Follower, Candidate, or Leader. Time is divided into arbitrary Terms (think of these as reigns of different kings). Each term starts with an election.
Here is how the transition of power happens step-by-step:
1. Heartbeat Timeout (The King Slumbers)
The Leader sends periodic "heartbeats" (empty AppendEntries RPCs) to all followers. If a follower stops receiving these heartbeats for a randomized period (typically between 150ms and 300ms), it assumes the leader has perished.
2. Campaign Season (The Candidate Emerges)
The follower increments its current term counter, transitions to the Candidate state, votes for itself, and sends a RequestVote RPC to all other nodes in the cluster.
3. Quorum Vote (The Barons Decant)
To win the crown, a candidate must receive votes from a strict majority (quorum) of the nodes in the cluster. For a cluster of size N, quorum is (N/2) + 1.
- In a 3-node cluster, quorum is 2.
- In a 5-node cluster, quorum is 3.
Implementing a Lightweight Leader Election in Go
Let's look at how we can implement a basic, lease-based leader election in Go. In production, you'd want to use a robust library or rely on a tool like etcd, but writing this from scratch helps demystify how these consensus engines actually work under the hood.
We will use a simulated "shared registry" (representing a state machine) with a TTL (Time To Live) lease to show how a node grabs and maintains its crown.
package main
import (
"context"
"fmt"
"math/rand"
"sync"
"time"
)
type Node struct {
ID string
IsLeader bool
Term int
mu sync.Mutex
}
type CentralRegistry struct {
mu sync.Mutex
ActiveLeader string
CurrentTerm int
LeaseExpires time.Time
}
func (r *CentralRegistry) TryAcquireLeadership(nodeID string, term int, duration time.Duration) bool {
r.mu.Lock()
defer r.mu.Unlock()
now := time.Now()
if now.After(r.LeaseExpires) || r.ActiveLeader == nodeID {
// Leadership lease expired or renewal by current leader
r.ActiveLeader = nodeID
r.CurrentTerm = term
r.LeaseExpires = now.Add(duration)
return true
}
return false
}
func runNode(ctx context.Context, node *Node, registry *CentralRegistry) {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
// Randomized election timeout to prevent split votes
electionTimeout := time.Duration(150+rand.Intn(150)) * time.Millisecond
leaseDuration := 3 * time.Second
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
node.mu.Lock()
if node.IsLeader {
// Heartbeat/Renewal
success := registry.TryAcquireLeadership(node.ID, node.Term, leaseDuration)
if success {
fmt.Printf("[Term %d] Node %s (Leader) successfully renewed its lease.\n", node.Term, node.ID)
} else {
fmt.Printf("[Term %d] Node %s lost its lease! Stepping down.\n", node.Term, node.ID)
node.IsLeader = false
}
} else {
// Follower check
registry.mu.Lock()
leaderAlive := time.Now().Before(registry.LeaseExpires)
activeLeader := registry.ActiveLeader
registry.mu.Unlock()
if !leaderAlive {
fmt.Printf("Node %s noticed the leader is dead. Initiating election...\n", node.ID)
node.Term++
// Artificial election delay
time.Sleep(electionTimeout)
success := registry.TryAcquireLeadership(node.ID, node.Term, leaseDuration)
if success {
node.IsLeader = true
fmt.Printf("👑 Node %s won the election for Term %d!\n", node.ID, node.Term)
}
} else {
fmt.Printf("Node %s acknowledges Leader %s is active.\n", node.ID, activeLeader)
}
}
node.mu.Unlock()
}
}
}
func main() {
rand.Seed(time.Now().UnixNano())
ctx, cancel := context.WithTimeout(context.Background(), 10 * time.Second)
defer cancel()
registry := &CentralRegistry{
LeaseExpires: time.Now().Add(-1 * time.Second), // Start expired
}
nodeA := &Node{ID: "Node-A", Term: 0}
nodeB := &Node{ID: "Node-B", Term: 0}
go runNode(ctx, nodeA, registry)
go runNode(ctx, nodeB, registry)
<-ctx.Done()
fmt.Println("Simulation finished.")
}
In this simplified Go code, you see the foundational elements of leader election: randomized timeouts to avoid collision, validation of lease terms, and stepping down when leadership cannot be verified.
Production-Grade Succession: How to Architect for Peace
If you're deploying distributed databases in AWS, GCP, or Azure, you shouldn't be writing your own consensus protocols unless you're a database engineer. Instead, you must architect your infrastructure so that your chosen engines (like PostgreSQL with Patroni, or CockroachDB) can resolve their succession crises quickly and safely.
1. The Odd-Number Rule
To survive a network partition, you must always run an odd number of voting nodes (typically 3 or 5). If you run 4 nodes, and a partition cuts them into two equal halves (2 and 2), neither side can achieve a strict majority (which requires 3 votes). Both sides freeze. By keeping your cluster size odd, you ensure that one side of a network partition will always have a majority and can elect a leader, while the minority side safely halts writes.
2. Multi-AZ Deployment
Do not put all your database nodes in the same Availability Zone (AZ). If that AZ goes dark, your entire kingdom falls. Distribute your 3-node cluster across 3 separate AZs. If AZ-1 suffers a catastrophic outage, AZ-2 and AZ-3 will form a majority, run an election, and keep your application online with minimal interruption.
Here is how a multi-AZ deployment handles an AZ failure gracefully:
+-------------------------------------------------------------+ | AWS Region (us-east-1) | | | | +----------------+ +----------------+ +----------------+ | | | AZ-1 | | AZ-2 | | AZ-3 | | | | [Node A: Dead] | | [Node B] | | [Node C] | | | +-------+--------+ +-------+--------+ +-------+--------+ | | | | | | | x (Disconnected) +---------+----------+ | | | | | [Quorum Reached: 2/3] | | [Node B Elected Leader] | +-------------------------------------------------------------+
3. Guard Rails Against Fencing (STONITH)
Sometimes, a failing leader refuses to step down because its JVM is garbage-collecting or its network stack is lagging. To prevent it from causing havoc when it wakes up, you need "fencing" mechanisms. In the infrastructure world, this is sometimes called STONITH ("Shoot The Other Node In The Head"). Using tools like Kubernetes persistent volume claims (PVCs) that can only be attached to one node at a time is a modern way of enforcing fencing.
Conclusion: The Price of Order
Whether you are ruling medieval England or managing a high-throughput microservices architecture on Kubernetes, the rules of succession remain the same. Chaos is the default state of the universe. Without strict, unambiguous rules for who holds authority—and how that authority is transferred when tragedy strikes—systems will always collapse into anarchy.
By leveraging consensus algorithms like Raft and configuring our cloud environments with multi-AZ, odd-numbered clusters, we ensure our applications can withstand the sudden death of their infrastructure "monarchs" without a single byte of data corruption.
How do you handle database failovers in your current stack? Have you ever had to clean up the mess of a split-brain database disaster? Let me know in the comments below!
Until next time, keep coding, keep learning, and keep your clusters in consensus.
— Alex