Low-stakes poll pipelines: durability, batching and idempotent tallying

This SysSeder guide examines an ordinary low-stakes polling pipeline.

A burst of responses is a useful engineering scenario without attaching it to a political candidate or an invented news event. This article is not a design for governmental elections, a secret ballot or identity assurance. It does not claim to protect democratic voting or to recount personal election-watching experience.

The scope is a small informational poll, such as choosing a documentation topic. Poll aggregation is not a distributed-consensus algorithm, and DAO governance or feature-flag coordination has additional requirements. We retain the Go/Redis/PostgreSQL design discussion but replace the unsafe runnable-looking service with a bounded, offline synthetic demonstration. No votes, accounts or infrastructure are accessed.

The Architecture of a Scalable Polling Engine

Simultaneous updates can contend on hot rows or saturate a connection pool. The outcome depends on workload, schema, capacity and backpressure; a relational database does not inevitably crash at the source’s invented traffic threshold. Benchmark representative authorized synthetic workloads before making capacity claims.

Separating intake from processing can smooth bursts, but moves the durability boundary. Decide exactly when a response is accepted, when it is durably recorded, and how a retry is recognized. The conceptual production path is:

Intake: authenticate and authorize an opaque principal, validate the poll and choice, bound the request, and durably enqueue a stable event ID. 202 Accepted means accepted for processing, not successfully counted. Provide a documented status mechanism; do not return “vote_registered” before recording.
Queue: Redis Lists and Streams have different delivery semantics. A simple destructive pop can lose a message if a worker fails before storage. Streams consumer groups track pending deliveries, but persistence, replication, trimming and recovery policy still determine loss behavior. No throughput or latency measurement is asserted.
Worker: process a bounded batch, retain event identities, and recover pending deliveries after failure. Acknowledge only after the database transaction is confirmed or an existing identical event has been reconciled. Retries must not add the same response twice.
Storage: commit a deduplicated event ledger and its tally effect in one transaction. Aggregate only newly accepted events. Batching can reduce overhead but adds latency, memory pressure and failure-recovery work; no orders-of-magnitude gain is guaranteed.

Conceptual architecture and acknowledgement boundary

Validated request → durable queue → bounded worker batch
Worker → transaction: unique event ledger + tally → confirmed commit
Confirmed commit or reconciled identical event → queue acknowledgement
Pending delivery after failure → recovery and idempotent retry
This is a design outline, not an implemented or tested deployment.

Offline Go intake model

Go’s net/http can support a service, but it is not inherently the perfect or fastest choice. A real HTTP implementation needs method and content-type handling, body limits, deadlines, authentication, authorization, TLS and overload behavior. Here we deliberately use only local strings and no listener or network client.

The next three Go blocks form one single-process teaching program in order. They use the standard library only: no go-redis/v9 dependency, Redis initialization or database connection. Principal IDs are synthetic values supplied by the fixture, never accepted from the JSON body. The program was not compiled or run here.

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "strings"
)

const maxBody = 4096
const maxBatch = 100
const maxLedger = 1000

type Event struct {
    ID, Poll, Choice, Principal string
}

type MemoryStore struct {
    events map[string]Event
    principals map[string]string
    totals map[string]int
}

func validID(s string) bool {
    if len(s) == 0 || len(s) > 64 { return false }
    for _, c := range s {
        if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' ||
             c >= '0' && c <= '9' || c == '-' || c == '_') { return false }
    }
    return true
}

The parser rejects oversized bodies, unknown or repeated JSON keys, trailing JSON and non-string values. It accepts only a fixed synthetic poll and its two choices. This illustrates input boundaries, not authentication. Asymptotic complexity of a queue operation does not establish end-to-end latency or durability.

func parseFixture(raw, principal string) (Event, error) {
    var e Event
    if len(raw) > maxBody || !validID(principal) {
        return e, errors.New("invalid input boundary")
    }
    d := json.NewDecoder(strings.NewReader(raw))
    token, err := d.Token()
    if err != nil || token != json.Delim('{') {
        return e, errors.New("expected one object")
    }
    fields := make(map[string]string)
    for d.More() {
        token, err = d.Token()
        if err != nil { return e, err }
        key, ok := token.(string)
        if !ok { return e, errors.New("invalid key") }
        if key != "event_id" && key != "poll_id" && key != "candidate_id" {
            return e, errors.New("unknown key")
        }
        if _, exists := fields[key]; exists {
            return e, errors.New("duplicate key")
        }
        var value string
        if err := d.Decode(&value); err != nil { return e, err }
        fields[key] = value
    }
    token, err = d.Token()
    if err != nil || token != json.Delim('}') {
        return e, errors.New("invalid object end")
    }
    if _, err = d.Token(); err != io.EOF {
        return e, errors.New("trailing data")
    }
    e = Event{fields["event_id"], fields["poll_id"], fields["candidate_id"], principal}
    if !validID(e.ID) || e.Poll != "demo-poll" ||
       (e.Choice != "topic-a" && e.Choice != "topic-b") {
        return Event{}, errors.New("invalid synthetic event")
    }
    return e, nil
}

Offline batch application and failure handling

The original worker has several independent loss paths: BRPOP removes the message before storage; malformed payloads are discarded; flushToDB only logs errors while its caller clears the buffer; and an uncertain commit can be counted twice on a blind retry. Simply increasing batch size fixes none of these.

A one-second blocking read inside the source loop can delay the claimed 500 ms flush. A production worker needs independent bounded receive/timer handling, cancellation, retry backoff and pending-message recovery. The offline model below instead demonstrates all-or-nothing batch application and duplicate handling; it is neither concurrent nor durable and makes no timer guarantee.

func (s *MemoryStore) applyBatch(batch []Event) error {
    if len(batch) > maxBatch { return errors.New("batch too large") }
    // Copy first: a rejected batch leaves the original maps unchanged.
    next := MemoryStore{map[string]Event{}, map[string]string{}, map[string]int{}}
    for k, v := range s.events { next.events[k] = v }
    for k, v := range s.principals { next.principals[k] = v }
    for k, v := range s.totals { next.totals[k] = v }
    for _, e := range batch {
        if !validID(e.ID) || !validID(e.Principal) || e.Poll != "demo-poll" ||
           (e.Choice != "topic-a" && e.Choice != "topic-b") {
            return errors.New("invalid event")
        }
        if old, exists := next.events[e.ID]; exists {
            if old != e { return errors.New("event ID conflict") }
            continue // Identical retry: no additional tally effect.
        }
        key := e.Poll + "/" + e.Principal
        if _, exists := next.principals[key]; exists {
            return errors.New("principal already responded")
        }
        if len(next.events) >= maxLedger { return errors.New("ledger full") }
        next.events[e.ID] = e
        next.principals[key] = e.ID
        next.totals[e.Choice]++
    }
    *s = next
    return nil
}

func main() {
    a, err := parseFixture(`{"event_id":"event-a","poll_id":"demo-poll","candidate_id":"topic-a"}`, "synthetic-a")
    if err != nil { panic(err) }
    b, err := parseFixture(`{"event_id":"event-b","poll_id":"demo-poll","candidate_id":"topic-b"}`, "synthetic-b")
    if err != nil { panic(err) }
    store := MemoryStore{map[string]Event{}, map[string]string{}, map[string]int{}}
    pending := []Event{a, b, a} // Includes an identical retry.
    if err := store.applyBatch(pending); err != nil {
        // Retain pending for explicit reconciliation; do not silently clear it.
        fmt.Println("Batch rejected:", err)
        return
    }
    pending = nil
    fmt.Println("Synthetic totals:", store.totals["topic-a"], store.totals["topic-b"])
    // By inspection the intended totals are 1 and 1, not an observed run.
}

SQL: atomic tally effects require an event ledger

Use a synthetic poll and choices, not a real election identifier. Retain each event’s identity through storage; aggregating away the identities before deduplication loses the information needed to distinguish retries from new responses.

PostgreSQL INSERT ... ON CONFLICT provides an atomic insert-or-update outcome for its row-level conflict operation. Adding an aggregate count is not idempotent: replay adds it again. Use unique ledger constraints and update totals only from newly inserted rows, within the same transaction. This does not create an atomic transaction across Redis and PostgreSQL.

The following SQL is a separate illustrative schema, not used by the offline Go program and not executed. It models permitted choices, opaque principals and one response per principal per poll. It is not identity proof or a secret ballot; operational retention and access rules are still required.

-- Illustration only: no database is created or changed here.
CREATE TABLE poll_choices (
    poll_id varchar(64) NOT NULL,
    candidate_id varchar(64) NOT NULL,
    PRIMARY KEY (poll_id, candidate_id)
);
CREATE TABLE poll_events (
    event_id varchar(64) PRIMARY KEY,
    poll_id varchar(64) NOT NULL,
    candidate_id varchar(64) NOT NULL,
    principal_id varchar(64) NOT NULL,
    UNIQUE (poll_id, principal_id),
    FOREIGN KEY (poll_id, candidate_id)
        REFERENCES poll_choices (poll_id, candidate_id)
);
CREATE TABLE poll_results (
    poll_id varchar(64) NOT NULL,
    candidate_id varchar(64) NOT NULL,
    vote_count bigint NOT NULL DEFAULT 0 CHECK (vote_count >= 0),
    PRIMARY KEY (poll_id, candidate_id),
    FOREIGN KEY (poll_id, candidate_id)
        REFERENCES poll_choices (poll_id, candidate_id)
);

Replace the original Go flushToDB with the following parameterized transaction pattern. Bind validated event, poll, choice and server-derived principal as $1–$4. The statement is a one-event illustration; a batch can group only new returned rows. If no row is inserted, read the existing record in a subsequent statement and compare the complete payload: identical retry, conflicting ID and repeated principal are different outcomes. This pattern assumes Read Committed isolation and retained immutable ledger entries. If reconciliation cannot establish an identical committed event, do not acknowledge: roll back and use a bounded recovery/retry policy. Handle database and serialization errors explicitly.

-- Parameterized illustration, not executed.
BEGIN ISOLATION LEVEL READ COMMITTED;
WITH accepted AS (
    INSERT INTO poll_events (event_id, poll_id, candidate_id, principal_id)
    VALUES ($1, $2, $3, $4)
    ON CONFLICT DO NOTHING
    RETURNING poll_id, candidate_id
)
INSERT INTO poll_results (poll_id, candidate_id, vote_count)
SELECT poll_id, candidate_id, 1 FROM accepted
ON CONFLICT (poll_id, candidate_id)
DO UPDATE SET vote_count = poll_results.vote_count + EXCLUDED.vote_count
RETURNING poll_id, candidate_id, vote_count;

-- If zero rows were returned, a subsequent parameterized SELECT compares
-- event_id and (poll_id, principal_id) records with the complete request.
-- Read Committed gives the subsequent statement a fresh snapshot.
-- If no identical committed event can be established, do not acknowledge;
-- roll back and reconcile/retry under a bounded policy. Ledger retention
-- must not delete or rewrite events needed for this retry window.
-- Do not report an altered payload as a successful retry.
-- The caller chooses COMMIT only for a verified accepted/identical outcome;
-- otherwise ROLLBACK. Check every statement and commit result.

Security and Anti-Abuse Considerations

Even an ordinary poll needs an abuse model. Bound requests and queues, enforce poll eligibility on the server, and minimize retained identifiers. These measures reduce specific risks; they do not establish election integrity or anonymity.

1. Rate limits and backpressure

Choose rate limits from capacity and abuse requirements rather than an arbitrary three-per-ten-seconds rule. Shared IPs can represent many legitimate users, and one client can have several IPs. Use authorized session/principal context where appropriate, avoid unnecessary IP retention, and bound queue depth. 429 Too Many Requests can include Retry-After; limits are not proof of one person per response.

2. Idempotency Keys and Double-Voting Prevention

A uniqueness constraint on poll and authenticated opaque principal enforces a policy about that principal, not about a unique human. Never trust a JSON user_id as identity. Bind retries to stable event IDs and compare payloads; keep ledger and tally effects atomic. Store no names, emails, IPs or political preferences for this teaching example, and do not describe a linkable ledger as anonymous.

3. Signature Verification

A signature or MAC authenticates possession of a key under a defined protocol, not a human voter or a fresh request. HMAC is a shared-secret MAC, not a public-key signature. Do not embed a shared signing secret in browser code. A legitimate machine-to-machine design also needs key management, exact signed-message framing, replay prevention, TLS and authorization; those operations are outside this article.

Conclusion

Decoupling, batching and atomic database operations are useful tools, not seamless-scaling guarantees. The essential review questions are where data becomes durable, how duplicate and conflicting events are handled, and what happens at every failure boundary.

No capacity result has been established for this design. A single in-memory process is neither a highly available service nor a durable store. A production proposal would require authorized load and failure tests, monitoring and recovery validation. Redis plus PostgreSQL does not automatically provide exactly-once delivery.

For discussion, use only sanitized synthetic cases: failure before commit, commit response lost, repeated delivery, a conflicting event ID, and queue saturation. What result should each case produce, and how will you verify it without collecting real responses?

Keep the demonstration offline and synthetic; do not repurpose it for consequential voting.

SysSeder — educational architecture review

Failure and recovery review

Never clear an uncommitted batch merely because an error was logged. If commit outcome is uncertain, reconcile stable event IDs before acknowledging; retries must preserve those IDs. Pending stream entries need recovery and a quarantine policy for invalid payloads. Do not trim required pending history. Bound receive waits, batch size, queue depth and retries; monitor lag and reject overload explicitly. Persistence settings determine the acknowledged-loss window. These are review requirements, not a tested implementation.