Hey everyone, Alex here from Coding with Alex. If you’ve scrolled through Hacker News lately, you might have spotted a headline that feels a bit more "art history" than "Kubernetes configuration": 1,300 Beautiful Wildlife Illustrations from the 19th Century Now Restored. It’s an incredible collection, showcasing the meticulous work of preserving delicate, century-old lithographs and watercolors.
But as developers, when we see a headline like that, our brains don't just appreciate the art—we start thinking about the infrastructure behind it. How do you ingest, process, store, and serve thousands of ultra-high-resolution, uncompressed archival images without melting your cloud budget or causing your users' browsers to choke?
Archival preservation is a massive engineering challenge. We are talking about source files that are often multi-gigabyte TIFFs, scanning artifacts that require automated color correction, metadata schema alignment (like Dublin Core), and the need to serve responsive, deeply zoomable images to web clients. Today, we are going to dive into how to build a modern, automated image processing and tile-serving pipeline using open-source tools, Node.js, and AWS. Grab a coffee, and let's build an archival asset pipeline.
The Architecture of an Archival Image Pipeline
When dealing with historical restorations, our architecture must handle two distinct phases: Ingestion & Batch Processing (asynchronous, heavy lifting) and Delivery (low-latency, dynamic resolution scaling). Here is a high-level overview of how a production-grade system looks:
[ High-Res Scan (TIFF) ] ──> S3 Upload (Ingest Bucket)
│
▼
S3 Event ──> AWS Lambda (sharp / libvips)
│
┌────────────────────────┴────────────────────────┐
▼ ▼
[ Deep-Zoom Tiles (DZI) ] [ Web-Ready WebP/AVIF ]
│ │
▼ ▼
S3 Delivery Bucket S3 Delivery Bucket
│ │
└────────────────────────┬────────────────────────┘
▼
CloudFront CDN
│
▼
[ OpenSeadragon Client ]
To make this work efficiently, we rely on three core pillars:
- Pyramid TIFFs & DZI (Deep Zoom Images): Instead of forcing a user to download a 500MB image to zoom in on a bird's feather, we slice the image into a multi-resolution pyramid of 256x256 pixel tiles.
- Libvips / Sharp: Standard image processing libraries like ImageMagick are notoriously memory-hungry because they load entire images into RAM.
libvipsis a stream-based alternative that processes massive files using almost no memory. - OpenSeadragon: An open-source, web-compatible viewer that dynamically requests only the specific tiles visible in the user's viewport.
Step 1: Setting Up the Batch Processor with Node.js and Sharp
Let's write a Node.js script that simulates our cloud worker. This script will take a massive raw TIFF file, apply some basic contrast normalization (essential for faded 19th-century paper), and output both a web-optimized WebP thumbnail and a Deep Zoom Image (DZI) folder structure.
First, make sure you have the necessary library installed:
npm install sharpNow, let's write our processing pipeline in processor.js:
const sharp = require('sharp');
const path = require('path');
const fs = require('fs').promises;
async function processArchivalImage(inputPath, outputDir, fileName) {
const startTime = Date.now();
console.log(`Starting processing for: ${fileName}`);
// Ensure our output directories exist
const dziOutputDir = path.join(outputDir, 'tiles', fileName);
const webOutputDir = path.join(outputDir, 'web');
await fs.mkdir(dziOutputDir, { recursive: true });
await fs.mkdir(webOutputDir, { recursive: true });
try {
// Initialize the Sharp pipeline with the raw high-res image
const pipeline = sharp(inputPath, { limitInputPixels: false }); // Disable limit for archival scans
// Step 1: Extract metadata to log dimensions
const metadata = await pipeline.metadata();
console.log(`Source Dimensions: ${metadata.width}x${metadata.height} px`);
// Step 2: Generate a standard web-optimized preview (WebP)
console.log("Generating web-friendly preview...");
await pipeline
.clone()
.resize(1600, null, { withoutEnlargement: true }) // Scale down to max 1600px width
.normalize() // Enhances contrast by stretching luminance
.webp({ quality: 80 })
.toFile(path.join(webOutputDir, `${fileName}.webp`));
// Step 3: Generate the Deep Zoom XML and Tile Pyramid
console.log("Generating Deep Zoom tile pyramid (this might take a few seconds)...");
await pipeline
.clone()
.normalize()
.tile({
size: 256,
overlap: 1,
layout: 'dz', // Deep Zoom format
container: 'fs' // Output directly to file system
})
.toFile(path.join(dziOutputDir, `${fileName}.xml`));
const duration = ((Date.now() - startTime) / 1000).toFixed(2);
console.log(`Success! Processing completed in ${duration}s`);
} catch (err) {
console.error(`Error processing image: ${err.message}`);
}
}
// Execute the pipeline
const input = path.join(__dirname, 'raw_input/wildlife_scan.tiff');
const output = path.join(__dirname, 'processed_output');
processArchivalImage(input, output, 'pheasant_illustration_1880');
Why this approach works
By using pipeline.clone(), we read the heavy raw file into memory stream buffers exactly once, fork the pipeline into two separate write operations (the preview and the tiles), and stream the output to disk. If you tried doing this with standard libraries, your serverless functions would constantly hit out-of-memory (OOM) limits.
Step 2: Deploying as a Serverless Worker
In a production system, you don't run this script manually. You deploy it to AWS Lambda or Google Cloud Functions. Since libvips is a compiled C library, deploying it inside a Node runtime on Lambda requires packaging the correct binaries for Linux/x64 or ARM64.
The easiest way to handle this in your infrastructure-as-code (like Terraform or Serverless Framework) is by using Docker containers for your Lambda functions, or by installing the architecture-specific dependencies during your CI/CD build phase:
npm install --platform=linux --arch=x64 sharpOnce deployed, you configure an S3 bucket event notification to trigger this Lambda function whenever a new .tiff file is uploaded to the /incoming prefix. The Lambda processes the file, dumps the tiles and WebP files into a public-facing, CloudFront-cached S3 bucket, and updates your application database with the metadata.
Step 3: Rendering Millions of Pixels on the Client Side
Now that our backend has sliced our 19th-century wildlife illustration into thousands of 256x256 pixel tiles, how do we show it to the user? If we try to load all of those tiles at once, the client's browser will crash. This is where OpenSeadragon comes in.
OpenSeadragon is an open-source, pure-JavaScript viewer for high-resolution zoomable images. It detects the user's viewport and zoom level, calculates exactly which tiles are needed, and fetches them on the fly.
Here is how easy it is to implement the frontend viewer:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Archival Viewer - Coding with Alex</title>
<!-- Include OpenSeadragon from CDN -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/openseadragon/4.1.0/openseadragon.min.js"></script>
<style>
body {
margin: 0;
background-color: #111;
color: #fff;
font-family: sans-serif;
}
#viewer-container {
width: 100vw;
height: 90vh;
background-color: #000;
}
.header {
height: 10vh;
display: flex;
align-items: center;
padding: 0 20px;
background: #1c1c1c;
}
</style>
</head>
<body>
<div class="header">
<h1 style="margin: 0; font-size: 1.5rem;">Restored 19th-Century Wildlife Viewer</h1>
</div>
<!-- This div is where the magic happens -->
<div id="viewer-container"></div>
<script>
// Initialize OpenSeadragon
const viewer = OpenSeadragon({
id: "viewer-container",
prefixUrl: "https://cdnjs.cloudflare.com/ajax/libs/openseadragon/4.1.0/images/",
tileSources: "https://my-cdn.com/tiles/pheasant_illustration_1880.xml", // Path to DZI XML generated by sharp
showNavigator: true,
navigatorPosition: "BOTTOM_RIGHT",
zoomInButton: "zoom-in-btn", // Custom controls can be mapped here
constrainDuringPan: true,
visibilityRatio: 1.0
});
</script>
</body>
</html>
How the Client-CDN-Storage loop works
When the page loads, OpenSeadragon fetches pheasant_illustration_1880.xml. This file contains metadata about the image's overall dimensions and tile format:
<Image xmlns="http://schemas.microsoft.com/deepzoom/2008" TileSize="256" Overlap="1" Format="jpg">
<Size Width="14000" Height="9800"/>
</Image>
Armed with this data, as the user pans around, OpenSeadragon calculates which level of zoom and which coordinate tile (e.g., /tiles/pheasant_illustration_1880_files/12/3_4.jpg) to fetch. CloudFront caches these static images, meaning subsequent loads are instant, bypassing your storage bucket entirely.
Performance Gains & Cloud Cost Optimization
Deploying this architecture yields massive improvements over standard image-hosting techniques:
- Reduced Bandwidth: Instead of downloading a massive 150MB JPEG or PNG, a user browsing the collection in deep detail typically downloads less than 3MB of active tiles.
- Better SEO & Core Web Vitals: By rendering a fast WebP preview for the initial page load and lazy-loading the OpenSeadragon canvas, your Largest Contentful Paint (LCP) remains under 1.2 seconds.
- Scale on Demand: S3 combined with CloudFront can handle millions of concurrent tile requests without a single backend server needing to scale or maintain state.
Wrapping Up & Over to You
Restoring physical art takes incredible skill and patience, but making that art accessible to the world requires elegant, efficient engineering. By combining the stream-processing power of libvips with the edge efficiency of CDNs and dynamic tile viewers, we can build web applications that handle massive assets with absolute ease.
Have you ever worked on high-resolution archiving or digital asset management pipelines? What tools did you use to handle the file sizes? Let’s talk about it in the comments below!
Until next time, keep coding.