Combining Hacker News and Wikipedia Pageviews: A Careful Content-Discovery Pipeline

We’ve all been there. It’s 11:30 PM, you have a production deployment running smoothly in the background, and you open Hacker News "just for five minutes." Two hours later, you’re deep in a rabbit hole reading about the history of the Lisp machine, Byzantine fault tolerance, or how the Roman aqueducts relate to modern microservices architectures.

As developers, we have a unique, insatiable appetite for niche, highly technical, and historically rich information. Wikipedia is often the destination for these deep dives. But how do we separate the signal from the noise? How do we discover the specific Wikipedia pages that actually capture the collective curiosity of the software engineering community?

OrangeCrumbs’ HN Wikipedia project was shared in a Show HN post titled “Discover Wikipedia articles popular on Hacker News.” It is one inspiration for a reading-discovery tool. The pipeline below is an illustrative design, not a verified account of that project’s license, ranking or internal implementation.

We will connect HN item retrieval, Wikipedia URL parsing, daily pageview enrichment and caching. The snippets illustrate these components; they are not a complete high-throughput service. Bounded work, retries, canonical-title resolution and a refresh scheduler still need implementation.

The Architecture of a Tech-Curiosity Pipeline

A structured discovery pipeline is easier to reason about than repeatedly scraping a rendered homepage. The design has three phases, with operational work kept outside the frontend’s page-load path:

Ingestion: Retrieve HN items and deliberately traverse story/comment relationships with bounded work.
Extraction and filtering: Read story URLs and parse comment HTML for href values, then validate and normalize supported Wikipedia URLs.
Aggregation and enrichment: Retrieve pageview series, preserve their project/date/traffic semantics and cache them for analysis.

Here is a conceptual flow of how data moves through this lightweight pipeline:


[ HN Firebase API ] ---> [ Bounded Ingestion Worker ]
                                |
                                v
                 [ HTML href extraction / URL parser ]
                                |
                                v
                   [ Wikimedia Pageviews API ]
                                |
                                v
               [ Redis Cache / SQLite DB ] ---> [ Frontend UI ]

The JavaScript examples below use a supported Node.js release with built-in fetch, URL parsing and timeout support. Offline checks covered selected parser/date/schema cases, not live API integration or performance.

Step 1: Mining the Hacker News API

HN exposes public data through its Firebase-backed API in near real time. Individual items and top lists are available over HTTP, while Firebase libraries can observe updates. The simple helpers below fetch snapshots; they do not implement streaming or a complex search engine.

Start by fetching top-list IDs and individual items. The top list can contain up to 500 entries, including jobs; it is not necessarily 500 stories. These helpers do not yet filter Wikipedia links or traverse comments.

// Use a supported Node.js release with built-in fetch; not a complete ingestion worker.
const HN_BASE_URL = 'https://hacker-news.firebaseio.com/v0';

async function requestJson(url, options = {}) {
  const response = await fetch(url, {
    ...options,
    signal: AbortSignal.timeout(10000)
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  return response.json();
}

async function getTopStoryIds() {
  const ids = await requestJson(`${HN_BASE_URL}/topstories.json`);
  if (!Array.isArray(ids)) throw new Error('Expected an ID array');
  return ids.filter(id => Number.isSafeInteger(id) && id > 0);
}

async function getItemDetails(id) {
  if (!Number.isSafeInteger(id) || id <= 0) throw new Error('Invalid item ID');
  return requestJson(`${HN_BASE_URL}/item/${id}.json`);
}

Story items may have a url; comments may have HTML in text. Traverse kids deliberately, handle null/deleted/dead items, and impose depth/work/concurrency limits. Extract href attributes with an HTML parser before passing individual URLs to the next helper. Those traversal and HTML-parser integrations are not included here.

Step 2: Parsing Wikipedia Slugs with Precision

A title is identifiable within a particular wiki, not globally across all language editions. URLs may use desktop or mobile hosts and titles may contain Unicode, parentheses or slashes. The following helper supports only /wiki/ URLs and an explicit English/German/Hebrew project allowlist; it does not parse arbitrary text or every URL form.

Use the URL parser to validate the host and path, then decode the title defensively. The result is a candidate project/title pair; resolve redirects and canonical titles through the appropriate API before merging records. Invalid percent encoding returns null rather than crashing the ingestion worker.

// Deliberately limited project allowlist; expand only with verified projects.
const SUPPORTED_PROJECTS = new Set([
  'en.wikipedia.org', 'de.wikipedia.org', 'he.wikipedia.org'
]);

// Input must be one extracted URL, not raw HTML or arbitrary prose.
function parseWikipediaUrl(rawUrl) {
  if (typeof rawUrl !== 'string') return null;
  let url;
  try { url = new URL(rawUrl); } catch { return null; }
  if (!['https:', 'http:'].includes(url.protocol)) return null;
  if (url.username || url.password || url.port) return null;
  const match = /^([a-z][a-z0-9-]*)\.(?:m\.)?wikipedia\.org$/.exec(url.hostname);
  if (!match || !url.pathname.startsWith('/wiki/')) return null;
  const project = `${match[1]}.wikipedia.org`;
  if (!SUPPORTED_PROJECTS.has(project)) return null;
  let slug;
  try { slug = decodeURIComponent(url.pathname.slice(6)).replace(/ /g, '_'); }
  catch { return null; }
  if (!slug || /[\u0000-\u001f\u007f]/.test(slug)) return null;
  return { lang: match[1], project, slug };
}

const example = parseWikipediaUrl(
  'https://en.wikipedia.org/wiki/C_(programming_language)#History'
);
// Expected: { lang: 'en', project: 'en.wikipedia.org',
//             slug: 'C_(programming_language)' }

Step 3: Enriching with the Wikimedia Pageviews API

Wikimedia’s public Pageviews API can enrich an article reference with aggregate traffic data. It does not tell us which visitors arrived from HN or whether a mention caused a spike. Use a descriptive User-Agent with valid operator contact information and follow the service’s access policy.

The helper below requests 30 completed UTC days ending yesterday, excludes identified automated traffic with user, and URI-encodes the entire title as one path component. It checks completeness rather than treating missing data as zero. Recent data can still be delayed. The preceding helpers and this code belong in the same module or need explicit imports. No live request was made for this revision.

// Uses SUPPORTED_PROJECTS and requestJson from the preceding examples.
function pageviewRequest(project, slug, now = new Date()) {
  if (!SUPPORTED_PROJECTS.has(project)) throw new Error('Unsupported project');
  if (typeof slug !== 'string' || !slug) throw new Error('Missing title');
  const end = new Date(Date.UTC(
    now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate() - 1
  ));
  const start = new Date(end);
  start.setUTCDate(start.getUTCDate() - 29);
  const format = date => date.toISOString().slice(0, 10).replace(/-/g, '');
  const dates = [];
  for (const date = new Date(start); date <= end; date.setUTCDate(date.getUTCDate() + 1)) {
    dates.push(format(date));
  }
  const title = encodeURIComponent(slug.replace(/ /g, '_'));
  const url = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/' +
    `${project}/all-access/user/${title}/daily/${format(start)}/${format(end)}`;
  return { url, dates };
}

async function getArticlePageviews(project, slug, userAgent, now = new Date(), fetchJson = requestJson) {
  if (typeof userAgent !== 'string' || !userAgent.trim()) {
    throw new Error('Supply your own descriptive User-Agent and contact information');
  }
  const { url, dates } = pageviewRequest(project, slug, now);
  const data = await fetchJson(url, { headers: { 'User-Agent': userAgent } });
  if (!Array.isArray(data.items)) throw new Error('Missing pageview data');
  const rows = new Map();
  for (const day of data.items) {
    if (typeof day.timestamp !== 'string' || !/^\d{10}$/.test(day.timestamp) ||
        !Number.isSafeInteger(day.views) || day.views < 0) {
      throw new Error('Invalid pageview row');
    }
    const date = day.timestamp.slice(0, 8);
    if (rows.has(date)) throw new Error('Duplicate pageview date');
    rows.set(date, day.views);
  }
  if (rows.size !== 30 || dates.some(date => !rows.has(date))) {
    throw new Error('Incomplete date range: keep missing data distinct from zero');
  }
  return dates.map(date => ({ date, views: rows.get(date) }));
}

Provide your own descriptive User-Agent and contact information: a relevant website, wiki user page or email address. Do not reuse an example author’s identity. Missing or generic identification can lead to blocking. Browser-based clients may need the documented Api-User-Agent alternative; this example is server-side Node.js.

Designing a High-Performance Caching Layer

Fetching every top item and its comments on each page load creates unnecessary request fan-out. HN’s API documentation currently states no rate limit; Wikimedia has a separate access policy. Cache results, refresh outside the rendering path, limit work and handle throttling or transient failures with bounded retries. The snippets do not yet implement that scheduler or retry policy.

Redis or SQLite can support this design. SQLite WAL permits readers and a writer to proceed concurrently, but only one writer at a time; it adds checkpointing and WAL/shared-memory files and requires same-host access rather than a shared network filesystem. It is not a guarantee of faster reads or zero overhead. Choose and measure for your workload.

Store each distinct article mention per HN item and project, including the HN publication timestamp separately from ingestion time. Cache pageviews by project, title, access method, agent type and day; aggregate date windows when reading. This allows multiple articles in one comment and avoids merging identical titles across languages. Schema creation alone does not enable WAL.

CREATE TABLE IF NOT EXISTS wikipedia_mentions (
    id INTEGER PRIMARY KEY,
    hn_item_id INTEGER NOT NULL,
    project TEXT NOT NULL,
    article_title TEXT NOT NULL,
    hn_score INTEGER,
    hn_published_at INTEGER NOT NULL,
    ingested_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
    UNIQUE (hn_item_id, project, article_title)
);

CREATE TABLE IF NOT EXISTS pageview_cache (
    project TEXT NOT NULL,
    article_title TEXT NOT NULL,
    access TEXT NOT NULL,
    agent TEXT NOT NULL,
    day TEXT NOT NULL,
    views INTEGER NOT NULL CHECK (views >= 0),
    fetched_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (project, article_title, access, agent, day)
);

A spike metric should compare the completed day of an HN post with an explicitly chosen preceding baseline that excludes that day. For an old post, fetch the matching historical window rather than today’s latest 30 days. A change from 200 to 15,000 daily views is a hypothetical example, not a measurement here. Specify how zero/missing baselines and incomplete days are handled; temporal correlation does not establish an HN referral or causal effect.

Why Tools Like This Make Us Better Developers

A discovery project can be designed to prioritize curiosity, history and technical depth rather than short-term engagement. That is an editorial design choice, not a measured statement about all social platforms or an automatic property of a recommendation algorithm.

Such a reading feed might lead you toward historical network protocols, biographies of computing pioneers or physical-infrastructure security. Treat those as possible reading directions, not findings from a dataset analysed in this article. The aim is to support slower, deliberate learning.

Conclusion: Build Your Own Discovery Engine

The HN Wikipedia project illustrates an appealing discovery idea. A separate implementation can combine public APIs, URL-aware parsing and carefully keyed caches. Reliability still depends on tested extraction, clear data semantics and operational limits, not simply the small size of the code.

Try adapting the design to a reading list you care about. If you add other communities or services, check their APIs, access conditions and data semantics separately rather than assuming the HN/Wikimedia rules apply.

What is the most interesting Wikipedia rabbit hole you have found through Hacker News? Let me know in the comments or share your thoughts on Twitter/X.

Until next time, happy coding!