Hey everyone, Alex here. Welcome back to another edition of Coding with Alex at sysseder.com.
If you've glanced at the tech news lately, you might have seen some heavy discussions circulating around the theme of "the dangerous delusion of modern warfare" and the vulnerabilities of highly centralized, digitized societies. While we usually stick to cleaner topics like refactoring legacy databases or optimizing Kubernetes ingress, this headline got me thinking about a critical, often ignored reality of modern software engineering: our absolute, fragile dependency on the global network.
As developers, we build systems under the comfortable assumption of "always-on" connectivity. We design microservices that constantly chat with one another, rely on third-party SaaS APIs for auth, and throw our hands up when AWS us-east-1 takes a nap. But what happens when the network doesn't just stutter, but completely goes dark? Whether it's a massive undersea cable failure, a severe state-sponsored cyberattack, or localized infrastructure collapse, "offline-first" isn't just a gimmick for mobile apps anymore. It's the ultimate tier of system resilience.
Today, we’re going to look at how to architect and build systems that can survive a total loss of external connectivity. We'll explore decentralized architectures, local-first state synchronization, and how to write code that degrades gracefully instead of throwing a 500 error and dying.
The Fallacy of the Constant Network
In distributed systems, we often talk about the "Fallacies of Distributed Computing." Number one on that list is: The network is reliable. Number two is: Latency is zero.
Yet, our modern stack is incredibly fragile. Think about a typical e-commerce checkout flow. To complete a single user transaction, your application might need to:
- Authenticate the user via Auth0 or Okta (External API)
- Query a cloud-managed PostgreSQL database (Cloud Network)
- Check inventory via an ERP service (Internal/External API)
- Process payment via Stripe (External API)
- Send a confirmation email via SendGrid (External API)
If any of those external connections fail, the entire transaction fails. If the wider internet drops, your business completely grinds to a halt. To survive extreme infrastructure failures, we need to design for autonomous local operation. This means designing systems that can process, queue, and eventually synchronize data once connectivity is restored.
Architectural Pattern: Event Sourcing and Outbox Pattern
To build a system that can run in isolation, you must decouple immediate synchronous actions from asynchronous eventual consistency. The best way to achieve this is by combining Event Sourcing with the Transactional Outbox Pattern.
Instead of trying to write directly to a remote database or call a remote API during a user request, you write the state change to a local, lightweight database (like SQLite or a local PostgreSQL instance) along with an "outbox" event in the same database transaction. A separate, local worker process then reads from this outbox table and attempts to forward the events to the outside world when a connection is available.
Here is a conceptual architecture of how a local-first node operates during an internet outage:
[ User Request ]
│
▼
┌────────────────────────────────────────┐
│ Local Service Node (Isolated) │
│ │
│ 1. Process Business Logic │
│ 2. Write to Local DB (ACID) │
│ ├── Update Local State │
│ └── Insert into "Outbox" Table │
└────────────────────────────────────────┘
│
▼ (Offline: Worker polls and retries)
[ Outbox Worker ] ──X──► [ Public Internet / Cloud API ]
Implementing a Resilient Outbox Worker in Go
Let's write some Go code to demonstrate how a local node can queue events to an outbox and safely retry publishing them without blocking the core user experience during a network partition.
First, let's define our database schema for the Outbox. If you are using SQLite for a local edge deployment, your schema might look like this:
CREATE TABLE outbox_events (
id TEXT PRIMARY KEY,
aggregate_type TEXT NOT NULL,
aggregate_id TEXT NOT NULL,
event_type TEXT NOT NULL,
payload TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
status TEXT DEFAULT 'PENDING', -- PENDING, PROCESSING, COMPLETED, FAILED
retry_count INTEGER DEFAULT 0
);Now, here is a resilient Go worker that processes these events. It uses exponential backoff to handle network instability, ensuring we don't spam a recovering network or crash our own local service.
package main
import (
"database/sql"
"fmt"
"log"
"math"
"time"
)
type Event struct {
ID string
Type string
Payload string
Retries int
}
// ProcessOutbox continuously polls the local DB for pending events
func ProcessOutbox(db *sql.DB) {
for {
events, err := fetchPendingEvents(db)
if err != nil {
log.Printf("Error fetching events: %v", err)
time.Sleep(5 * time.Second)
continue
}
for _, event := range events {
err := publishToCloud(event)
if err != nil {
log.Printf("Failed to publish event %s: %v. Backing off...", event.ID, err)
handleFailure(db, event)
} else {
log.Printf("Successfully published event %s", event.ID)
markAsCompleted(db, event.ID)
}
}
time.Sleep(2 * time.Second) // Poll interval
}
}
func publishToCloud(event Event) error {
// Simulate network call. In a real app, this would be an HTTP/gRPC call.
// If the network is down, this will return an error.
isNetworkUp := false
if !isNetworkUp {
return fmt.Errorf("network connection timed out")
}
return nil
}
func fetchPendingEvents(db *sql.DB) ([]Event, error) {
rows, err := db.Query("SELECT id, event_type, payload, retry_count FROM outbox_events WHERE status = 'PENDING' LIMIT 10")
if err != nil {
return nil, err
}
defer rows.Close()
var events []Event
for rows.Next() {
var e Event
if err := rows.Scan(&e.ID, &e.Type, &e.Payload, &e.Retries); err != nil {
return nil, err
}
events = append(events, e)
}
return events, nil
}
func handleFailure(db *sql.DB, event Event) {
event.Retries++
// Calculate exponential backoff: 2^retries seconds (capped at 1 hour)
backoffSec := math.Min(math.Pow(2, float64(event.Retries)), 3600)
nextAttempt := time.Now().Add(time.Duration(backoffSec) * time.Second)
log.Printf("Scheduling retry for event %s in %.0f seconds", event.ID, backoffSec)
_, err := db.Exec(
"UPDATE outbox_events SET retry_count = ?, status = 'PENDING' WHERE id = ?",
event.Retries, event.ID,
)
if err != nil {
log.Printf("Failed to update retry state in local DB: %v", err)
}
// Sleep to prevent tight-looping during outage
time.Sleep(time.Duration(backoffSec) * time.Second)
}
func markAsCompleted(db *sql.DB, id string) {
_, err := db.Exec("UPDATE outbox_events SET status = 'COMPLETED' WHERE id = ?", id)
if err != nil {
log.Printf("Failed to mark event %s as completed: %v", id, err)
}
}Why This Matters During a Crisis
In the code above, if the connection to the cloud completely vanishes, our application doesn't fail. Users can still perform actions locally (such as checking in, logging metrics, or saving local reports). The local node records these actions as absolute truth in its local SQLite database. Once the network partition heals—whether that takes 5 minutes or 5 days—the outbox worker catches up, shipping state modifications back to the central hub.
Handling Conflict: The Pain of eventual Consistency
Of course, offline-first architectures aren't a free lunch. When you allow multiple isolated nodes to write data locally and sync later, you inevitably run into conflict. What happens if two nodes edit the same record while offline?
To solve this without a central coordinator, we must look to mathematical structures designed for distributed networks:
1. Conflict-Free Replicated Data Types (CRDTs)
CRDTs are data structures that can be updated independently and concurrently without coordination, and they are mathematically guaranteed to converge to the same state. If your application keeps track of counters, sets, or state logs, CRDTs are a lifesaver.
2. Last-Write-Wins (LWW) vs. MVCC
For simple key-value stores, you can use a Last-Write-Wins strategy using hybrid logical clocks (HLC) to determine which write is the "latest." However, in mission-critical systems, you should preserve both versions of the conflict and let the user or a business-logic reconciliation engine resolve the fork manually.
Practical Checklist for "Zero-Network" Preparedness
If you want to start hardening your systems against localized or global network failures today, here is a practical checklist you can implement in your sprints:
- Implement aggressive caching with stale-while-revalidate policies: Ensure your application can load and display last-known-good data if its primary APIs are unreachable.
- Gracefully degrade auth: Instead of blocking users when your identity provider goes down, configure your gateways to accept expired JWTs for a grace period, or fall back to pre-shared offline session keys for critical operators.
- Keep static assets local: Avoid loading essential CSS, JS, or fonts from external CDNs. If public CDNs are blocked or down, your application should still render perfectly from its own host.
- Use local message brokers: Run local instances of RabbitMQ, Mosquitto (MQTT), or NATS on your edge servers to handle inter-process communication locally, rather than sending messages up to a cloud-managed queue.
Wrapping Up: Build for Chaos
As the systems that govern our world grow increasingly complex, the potential for catastrophic failure grows with them. Designing software that assumes a persistent, high-speed, global connection is no longer just optimistic—it's a security and operational risk.
By shifting our mental model to "local-first, cloud-second," we build software that is not only faster and more responsive for everyday users, but resilient enough to survive when the modern network infrastructure falters.
What about you? How does your team handle network degradation? Have you experimented with SQLite edge synchronization, or do you rely heavily on active cloud connectivity? Let’s chat in the comments below!
Until next time, keep your systems decoupled and your backups local. Happy coding!