Hey everyone, Alex here. Welcome back to another edition of Coding with Alex on sysseder.com.
If you've been hanging around the systems engineering space lately, you've probably noticed a quiet but powerful shift. For years, when we needed a fast, in-memory cache or key-value store, we defaulted to Redis or Memcached. But as modern NVMe drives approach RAM-like speeds, and as CPU cache coherence becomes the primary bottleneck in highly concurrent systems, the architectural trade-offs of the past are being rewritten.
Enter Eigendrum. It’s a highly experimental, ultra-fast, open-source key-value store built in Rust that leverages memory-mapped files (mmap), lock-free data structures, and zero-copy deserialization to achieve jaw-dropping throughput. If you are interested in low-latency systems, database internals, or how to squeeze every drop of performance out of modern hardware, Eigendrum is a goldmine of systems design patterns. Today, we are going to tear it open, look at how it works under the hood, and build our own educational micro-version of a memory-mapped, lock-free key-value store in Rust.
Why the "Mmap + Lock-Free" Pattern Matters Right Now
Before we dive into the code, let's talk about the traditional database bottleneck: serialization and context switching. In a traditional database architecture, a read request looks like this:
- The kernel reads data from the disk into the OS page cache.
- The database engine copies that data from the page cache into user-space memory.
- The engine deserializes the raw bytes into language-level structures (like a struct or an object).
- Locks are acquired and released to ensure thread safety while accessing these structures.
This process is riddled with CPU cache misses, memory allocation overhead, and thread contention. Eigendrum bypasses almost all of this by mapping files directly into the virtual address space of the process using the mmap system call. This allows the OS to handle paging lazily and transparently. Combined with zero-copy deserialization (using libraries like bincode or flatbuffers) and atomic pointer swapping, reads become practically free—requiring zero memory copies and zero locks.
The Architecture of a High-Performance Mapped Store
To understand Eigendrum, we need to understand how it organizes data on disk and in memory. The architecture can be broken down into three major components:
1. The Virtual Address Space (The Arena)
Instead of allocating memory on the heap using standard allocators, Eigendrum maps a large file on disk directly into virtual memory. This is our "Arena". The operating system maps pages of this file into physical RAM only when they are accessed (page faults). When we write to this memory, the OS dirty pages are periodically flushed back to the disk asynchronously by the kernel daemon (pdflush/flush), or forced manually via msync.
2. The Lock-Free Index (Atomic Radix Tree or Hash Table)
To find where a key is located within our memory-mapped file, we need an index. Eigendrum uses a lock-free indexing mechanism. Instead of wrapping the entire index in a heavy Mutex or RwLock, it uses atomic pointers. Readers can read the index concurrently without blocking, while writers use atomic Compare-And-Swap (CAS) operations to update pointers to new versions of the data.
3. Append-Only Log Structuring
To maintain write performance and prevent fragmentation, Eigendrum uses a log-structured layout within the mapped file. Writes are always appended to the end of the active segment. When a key is updated, we don't overwrite the old data in place; we append the new value to the end of the log and atomically update the index pointer to point to the new offset. A background "garbage collection" or compaction thread later reclaims space from abandoned offsets.
Building a Lightweight Memory-Mapped Store in Rust
Let's roll up our sleeves and write some Rust. We are going to build a simplified version of this architecture. We will use the memmap2 crate to handle safe memory-mapping and crossbeam-utils for atomic operations.
First, let's set up our Cargo.toml dependencies:
[dependencies]
memmap2 = "0.9.4"
libc = "0.2"
crossbeam-utils = "0.8"
Designing our Storage Layout
We will define a simple on-disk representation for our key-value pairs. Each record appended to our memory-mapped file will have a fixed-size header followed by the variable-length key and value.
use std::fs::{OpenOptions, File};
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
use memmap2::{MmapMut, MmapOptions};
// Layout: [8-byte magic] [4-byte key_len] [4-byte val_len] [Key bytes...] [Value bytes...]
const HEADER_SIZE: usize = 16;
const FILE_SIZE: usize = 100 * 1024 * 1024; // 100 MB Arena
pub struct MappedStore {
mmap: MmapMut,
write_offset: AtomicU64,
file: File,
}
Initializing the Memory Map
Now, let's implement the initialization logic. We open (or create) a file, truncate it to our desired arena size, and map it into memory as mutable.
impl MappedStore {
pub fn new<P: AsRef<Path>>(path: P) -> std::io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open(path)?;
// Allocate disk space for our arena
file.set_len(FILE_SIZE as u64)?;
// Map the file into virtual memory
let mmap = unsafe { MmapOptions::new().map_mut(&file)? };
// For simplicity, we start writing at offset 0.
// In a production engine, we would parse the existing file to find the end of the log.
let write_offset = AtomicU64::new(0);
Ok(MappedStore {
mmap,
write_offset,
file,
})
}
}
Implementing Lock-Free Concurrent Appends
The beauty of this architecture is that multiple threads can write concurrently without holding an expensive lock on the entire database. We achieve this by using an atomic fetch_add on our write_offset. Each thread safely reserves its own unique slice of the memory map to write its payload.
impl MappedStore {
pub fn append(&self, key: &[u8], value: &[u8]) -> Result<usize, &'static str> {
let entry_size = HEADER_SIZE + key.len() + value.len();
// Atomically reserve our slot in the memory map
let start_offset = self.write_offset.fetch_add(entry_size as u64, Ordering::SeqCst) as usize;
if start_offset + entry_size > FILE_SIZE {
return Err("Out of mapped space! Compaction required.");
}
// Get a direct pointer to our reserved slice of the mmap.
// Even though mmap is shared, this raw pointer access is safe because
// our atomic fetch_add guarantees exclusive access to this byte range.
unsafe {
let mmap_ptr = self.mmap.as_ptr() as *mut u8;
let dest = std::slice::from_raw_parts_mut(mmap_ptr.add(start_offset), entry_size);
// Write Magic Number (0xDEADBEEF)
dest[0..4].copy_from_slice(&0xDEADBEEFu32.to_le_bytes());
// Write Key & Value Lengths
dest[4..8].copy_from_slice(&(key.len() as u32).to_le_bytes());
dest[8..12].copy_from_slice(&(value.len() as u32).to_le_bytes());
// Write payload
let key_start = HEADER_SIZE;
let key_end = key_start + key.len();
dest[key_start..key_end].copy_from_slice(key);
dest[key_end..key_end + value.len()].copy_from_slice(value);
}
Ok(start_offset)
}
}
Zero-Copy Reads
Reading from our store is extraordinarily fast. Because the file is memory-mapped, we don't need to read bytes into a buffer. We can return a slice pointing directly to the virtual memory managed by the OS.
impl MappedStore {
pub fn read_at(&self, offset: usize) -> Result<(&[u8], &[u8]), &'static str> {
if offset + HEADER_SIZE > FILE_SIZE {
return Err("Offset out of bounds");
}
unsafe {
let mmap_ptr = self.mmap.as_ptr();
// Read magic number to verify block integrity
let mut magic_bytes = [0u8; 4];
magic_bytes.copy_from_slice(std::slice::from_raw_parts(mmap_ptr.add(offset), 4));
let magic = u32::from_le_bytes(magic_bytes);
if magic != 0xDEADBEEF {
return Err("Invalid block corruption or empty space");
}
// Extract lengths
let mut key_len_bytes = [0u8; 4];
key_len_bytes.copy_from_slice(std::slice::from_raw_parts(mmap_ptr.add(offset + 4), 4));
let key_len = u32::from_le_bytes(key_len_bytes) as usize;
let mut val_len_bytes = [0u8; 4];
val_len_bytes.copy_from_slice(std::slice::from_raw_parts(mmap_ptr.add(offset + 8), 4));
let val_len = u32::from_le_bytes(val_len_bytes) as usize;
// Generate zero-copy references directly pointing to our memory-mapped file!
let key = std::slice::from_raw_parts(mmap_ptr.add(offset + HEADER_SIZE), key_len);
let val = std::slice::from_raw_parts(mmap_ptr.add(offset + HEADER_SIZE + key_len), val_len);
Ok((key, val))
}
}
}
Putting It All Together with a Lock-Free Index
To make this functional, we need a way to look up where our keys are. In a complete implementation like Eigendrum, this is done via a highly optimized concurrent map. Let's look at how we can integrate a simple concurrent index to coordinate our storage engine:
use std::collections::ConcurrentMap; // Conceptual
use std::sync::Arc;
use crossbeam_utils::thread;
fn main() {
// Initialize our memory-mapped file
let store = Arc::new(MappedStore::new("demo.db").expect("Failed to map file"));
// We'll use an atomic pointer reference to map keys to their offsets.
// For simplicity, let's simulate concurrent writer threads writing data.
let mut handles = vec![];
for i in 0..4 {
let store_clone = Arc::clone(&store);
let handle = std::thread::spawn(move || {
let key = format!("user_session_{}", i);
let val = format!("session_data_payload_for_user_{}_containing_metadata", i);
let offset = store_clone.append(key.as_bytes(), val.as_bytes()).unwrap();
println!("Thread {} wrote {} bytes at offset {}", i, key.len() + val.len() + HEADER_SIZE, offset);
// In a real application, you would now update your Lock-Free Radix Tree:
// index.insert(key, offset);
offset
});
handles.push(handle);
}
// Join threads and read back the data zero-copy style
for handle in handles {
let offset = handle.join().unwrap();
let (key, val) = store.read_at(offset).unwrap();
println!("Successfully read back: Key='{}', Val='{}' directly from mapped physical memory!",
String::from_utf8_lossy(key),
String::from_utf8_lossy(val)
);
}
}
The Trade-Offs: When NOT to Use This Architecture
As impressive as Eigendrum’s architecture is, software engineering is always a game of compromises. There are several critical trade-offs you must consider before ripping out Redis in favor of a memory-mapped KV store:
1. Virtual Memory Address Space Exhaustion
Because you are mapping files directly to virtual memory, you are limited by the system's address space. While 64-bit operating systems provide a massive address space (typically 128 TB on modern x86_64), running out of address space is still a concern if you are managing petabyte-scale data sets or dealing with highly fragmented processes.
2. The "I/O Freeze" Page Fault Risk
When you read from a memory-mapped file, the OS assumes the page is in memory. If it’s not, the CPU triggers a major page fault, blocking the execution thread while the kernel fetches the page from disk. In high-throughput, single-threaded event loops (like Node.js or Redis), a single page fault can block the entire server. Eigendrum mitigates this by utilizing worker thread pools to handle reads and exploiting OS-level flags like MADV_WILLNEED to aggressively pre-fetch pages.
3. Crash Consistency and Corruptions
If your application crashes when writing to standard heap memory, your operating system keeps running, and your files are safe. If your application crashes while mutating a memory map, or if the system suffers a sudden power loss, you risk writing partially formed or corrupted records directly to disk. Developing a robust, write-ahead logging (WAL) mechanism or transactional commit protocol is critical to ensuring database durability.
Conclusion and Next Steps
The engineering patterns behind Eigendrum demonstrate how aligning software architecture with operating system capabilities can lead to massive performance gains. By pairing mmap with lock-free structures, we bypass user-space buffers, minimize thread synchronization bottlenecks, and let the Linux kernel do what it does best: manage caching and I/O pacing.
If you're building next-generation telemetry engines, fast caching layers, or edge-optimized embedded databases, digging into these patterns will make you a better systems engineer.
What are your thoughts? Have you ever experimented with memory-mapped files or lock-free storage in Rust? What challenges did you face with crash consistency? Let’s chat in the comments below!
If you enjoyed this deep dive, don’t forget to subscribe to the "Coding with Alex" newsletter and share this post with your fellow systems developers. Until next time, happy coding!