Hey everyone, Alex here. Welcome back to another edition of Coding with Alex on sysseder.com.
If you've glanced at tech news or social media recently, you might have seen the name "Count Binface" trending yet again. For those outside the UK, Count Binface is a satirical political candidate—a self-proclaimed space warrior with a bin for a head—who regularly runs in high-profile elections, challenging prime ministers and mayors. It's a glorious, deeply democratic tradition of British eccentricity. But as I was watching the election results trickle in, my developer brain did what it always does: it started thinking about the underlying infrastructure of democracy, decision-making, and public opinion. Specifically, how do we build digital voting, polling, and consensus systems that are secure, highly available, and resilient to massive traffic spikes?
Whether you are building a real-time polling app for a live stream, a governance tool for a DAO, a feature-flagging system that relies on distributed consensus, or a classic upvoting engine like Hacker News, handling concurrent state updates without losing data is a classic distributed systems problem. Today, we're going to step away from the political theater and dive deep into the engineering theater. We'll explore how to design and build a high-throughput, concurrent polling engine using Go, Redis, and PostgreSQL. Grab your coffee (or your space helmet), and let's get coding.
The Architecture of a Scalable Polling Engine
When designing a system to ingest votes or poll responses, you face a classic write-heavy workload. If 100,000 users click "Vote for Count Binface" at the exact same millisecond, writing directly to a traditional relational database will quickly lead to row locking, connection pool exhaustion, and eventually, a system-wide crash.
To handle this, we need to decouple our write path from our storage path. Here is the architectural pattern we will implement:
- The Ingestion Layer (Go): A lightweight, stateless HTTP API built in Go. It validates incoming votes and immediately pushes them to an in-memory queue. It doesn't wait for database confirmation; it simply returns a
202 Acceptedstatus to the client. - The Buffer Layer (Redis): We use Redis as an in-memory data store. Specifically, we'll use Redis Streams or Lists to act as a highly performant message broker that can handle hundreds of thousands of operations per second with sub-millisecond latency.
- The Worker Pool (Go): A background consumer group that pulls batches of votes from Redis, aggregates them in memory, and writes them to the database in bulk.
- The Storage Layer (PostgreSQL): Our source of truth. By batching writes (e.g., updating database totals once every second or every 1,000 votes), we reduce our database transaction load by orders of magnitude.
Architecture Diagram
[ Client App ] --( HTTP POST /vote )--> [ Go API Ingestor ]
|
( LPUSH / Redis Stream )
v
[ Redis Buffer ]
|
( Batch Consumer )
v
[ Go Worker Pool ] --( Bulk UPSERT )--> [ PostgreSQL ]
Setting Up the Go API Ingestor
Go is the perfect language for this use case because of its lightweight concurrency model (goroutines) and excellent low-level network performance. Let's start by writing our HTTP handler to accept votes. We'll use the standard library's net/http package to keep dependencies minimal and performance maximum.
First, let's define our payload structure and the basic setup for our Redis client. We will use the popular go-redis/v9 library.
package main
import (
"context"
"encoding/json"
"log"
"net/http"
"github.com/redis/go-redis/v9"
)
var (
ctx = context.Background()
redisClient *redis.Client
)
type Vote struct {
PollID string `json:"poll_id"`
CandidateID string `json:"candidate_id"`
UserID string `json:"user_id"`
}
func initRedis() {
redisClient = redis.NewClient(&redis.Options{
Addr: "localhost:6379",
})
}
Now, let's write the handler that receives the vote, validates it briefly, and pushes it into a Redis List acting as our queue. Pushing to a Redis List using LPUSH is an $O(1)$ operation, making it incredibly fast.
func voteHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
var v Vote
err := json.NewDecoder(r.Body).Decode(&v)
if err != nil || v.PollID == "" || v.CandidateID == "" || v.UserID == "" {
http.Error(w, "Bad request: invalid vote data", http.StatusBadRequest)
return
}
// Serialize the vote payload
payload, err := json.Marshal(v)
if err != nil {
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Push the vote to the Redis queue named "votes_queue"
err = redisClient.LPush(ctx, "votes_queue", payload).Err()
if err != nil {
log.Printf("Failed to push vote to Redis: %v", err)
http.Error(w, "Service temporarily unavailable", http.StatusServiceUnavailable)
return
}
// Respond immediately to the client
w.WriteHeader(http.StatusAccepted)
w.Write([]byte(`{"status":"vote_registered"}`))
}
Building the Batch Worker Daemon
If we wrote every single vote from the queue straight to PostgreSQL one by one, we would quickly run into disk I/O bottlenecks. Instead, we need a worker daemon that runs in the background, pulls batches of votes from Redis, aggregates the counts in memory, and performs a bulk update in PostgreSQL.
Let's look at how we can implement this aggregation logic in Go. We will read up to 100 votes at a time or wait a maximum of 500 milliseconds before flushing our buffer to the database. This pattern is often called "micro-batching."
package main
import (
"context"
"database/sql"
"log"
"time"
_ "github.com/lib/pq"
)
type VoteBatcher struct {
db *sql.DB
redisClient *redis.Client
batchSize int
flushTimer time.Duration
}
func NewVoteBatcher(db *sql.DB, rClient *redis.Client, size int, flush time.Duration) *VoteBatcher {
return &VoteBatcher{
db: db,
redisClient: rClient,
batchSize: size,
flushTimer: flush,
}
}
func (vb *VoteBatcher) Start(ctx context.Context) {
ticker := time.NewTicker(vb.flushTimer)
defer ticker.Stop()
// In-memory aggregator map[PollID]map[CandidateID]Count
localBuffer := make(map[string]map[string]int64)
count := 0
for {
select {
case <-ctx.Done():
log.Println("Shutting down worker, flushing remaining votes...")
vb.flushToDB(localBuffer)
return
case <-ticker.C:
if count > 0 {
log.Println("Time-based flush triggered...")
vb.flushToDB(localBuffer)
localBuffer = make(map[string]map[string]int64)
count = 0
}
default:
// Pop a vote from Redis. BRPop blocks for a short time to avoid tight looping
res, err := vb.redisClient.BRPop(ctx, 1*time.Second, "votes_queue").Result()
if err == redis.Nil {
continue // Queue is empty, loop again
} else if err != nil {
log.Printf("Redis error: %v", err)
continue
}
// Parse the vote payload
var v Vote
if err := json.Unmarshal([]byte(res[1]), &v); err != nil {
log.Printf("Failed to unmarshal vote: %v", err)
continue
}
// Aggregate in memory
if _, exists := localBuffer[v.PollID]; !exists {
localBuffer[v.PollID] = make(map[string]int64)
}
localBuffer[v.PollID][v.CandidateID]++
count++
// Trigger bulk flush if batch size is reached
if count >= vb.batchSize {
log.Println("Batch-size flush triggered...")
vb.flushToDB(localBuffer)
localBuffer = make(map[string]map[string]int64)
count = 0
}
}
}
}
The SQL Secret: Atomic UPSERTs
Now that we have aggregated our votes in memory (e.g., 500 votes for candidate_id: "count-binface" in poll_id: "london-mayor-2024"), we need to write them to PostgreSQL.
Instead of running multiple UPDATE statements or selecting the row first to check if it exists, we will use an atomic INSERT ... ON CONFLICT statement (also known as an UPSERT). This ensures thread safety, avoids race conditions, and is incredibly efficient.
First, let's look at the database schema we'd use:
CREATE TABLE poll_results (
poll_id VARCHAR(255) NOT NULL,
candidate_id VARCHAR(255) NOT NULL,
vote_count BIGINT DEFAULT 0 NOT NULL,
PRIMARY KEY (poll_id, candidate_id)
);
Now, let's implement the flushToDB method in Go. We will use a database transaction to ensure that either all updates in the batch succeed, or none do.
func (vb *VoteBatcher) flushToDB(buffer map[string]map[string]int64) {
if len(buffer) == 0 {
return
}
tx, err := vb.db.Begin()
if err != nil {
log.Printf("Failed to begin transaction: %v", err)
return
}
defer tx.Rollback()
// Prepare the upsert statement
stmt, err := tx.Prepare(`
INSERT INTO poll_results (poll_id, candidate_id, vote_count)
VALUES ($1, $2, $3)
ON CONFLICT (poll_id, candidate_id)
DO UPDATE SET vote_count = poll_results.vote_count + EXCLUDED.vote_count;
`)
if err != nil {
log.Printf("Failed to prepare statement: %v", err)
return
}
defer stmt.Close()
// Execute upserts for each aggregated candidate
for pollID, candidates := range buffer {
for candidateID, count := range candidates {
_, err := stmt.Exec(pollID, candidateID, count)
if err != nil {
log.Printf("Failed to execute upsert: %v", err)
return // Transaction will rollback automatically
}
}
}
if err := tx.Commit(); err != nil {
log.Printf("Failed to commit transaction: %v", err)
return
}
log.Printf("Successfully flushed batch to the database.")
}
Security and Anti-Abuse Considerations
In a real-world scenario—and definitely in one involving political candidates like Count Binface—you will have malicious actors, bots, and script kiddies trying to flood your API to rig the polls. A resilient polling engine must address this on multiple fronts:
1. Rate Limiting with Redis Token Bucket
Before pushing a vote to the queue, check the user's IP or user session using a rate limiter. Redis is perfect for implementing a sliding-window or token-bucket rate-limiting algorithm. If a specific IP address tries to vote more than 3 times in 10 seconds, immediately drop the request and return a 429 Too Many Requests.
2. Idempotency Keys and Double-Voting Prevention
To prevent users from voting multiple times, you need a mechanism to verify identity. If you require registration, you can use a unique composite constraint in PostgreSQL on (poll_id, user_id). However, writing individual user votes to database tables requires a different schema design than our aggregated table. If you must track who voted, write individual vote transactions to a ledger table, and use database triggers or offline jobs to update the totals.
3. Signature Verification
If votes are coming from third-party client apps or hardware terminals, use cryptographic signatures (such as HMAC keys or Ed25519 public/private keys) to sign the payload. The API ingestor should verify the signature before processing the vote.
Conclusion
Building systems that scale seamlessly under sudden, massive interest—whether it's for a satirical space warrior, a flash-sale e-commerce platform, or a sudden surge in real-time telemetry data—comes down to a few fundamental design patterns: decoupling writes, micro-batching, in-memory aggregation, and atomic database operations.
By pairing Go's high-concurrency primitives with Redis's ultra-low latency and PostgreSQL's reliable ACID compliance, we can build an engine capable of handling millions of votes on a single, modestly sized cloud instance.
What are your thoughts on this architecture? Have you ever had to build a high-throughput polling or telemetry pipeline? How would you handle double-voting at scale without killing database performance? Let me know in the comments below!
As always, keep coding, keep building, and may the best candidate (human or alien) win.
— Alex