Hey everyone, welcome back to Coding with Alex!
If you scrolled through Hacker News today, you probably saw a fascinating headline: "An Englishwoman who sketched India before photography took hold." It's a gorgeous piece detailing the work of Charlotte Canning, an artist in the 1850s who captured highly detailed landscapes, architecture, and daily life in India using nothing but watercolors and graphite.
Now, you might be asking: "Alex, this is a tech blog. Why are we talking about 19th-century watercolor sketches?"
Because as software engineers, web developers, and database architects, we are currently facing a massive, industry-wide challenge: How do we index, search, and preserve non-textual historical archives at scale?
If you have thousands of fragile, scanned historical sketches with zero metadata, a standard relational database and elasticsearch keyword search are completely useless. You can't search for "Mughal archway in the sunset" if nobody has manually tagged that image. Today, we are going to bridge the gap between 1850s art and 2024 engineering. We’ll build a complete, production-ready pipeline to ingest historical sketches, generate high-quality vector embeddings using open-source Vision-Language Models (VLMs), store them in a vector database, and build a semantic search API. Let’s dive in!
The Architectural Blueprint: Semantic Image Retrieval
To make historic sketches searchable, we can't rely on basic OCR (Optical Character Recognition) because there is no text. Instead, we use Vector Search (or Semantic Search).
We will pass our scanned images through a pre-trained deep learning model (CLIP - Contrastive Language-Image Pre-training) developed by OpenAI. CLIP is unique because it embeds both images and text into the same vector space. This means an image of a sketch and the English phrase describing that sketch will have vector representations that are incredibly close to each other mathematically (measured by cosine similarity).
Here is how our pipeline looks:
[ Historical Sketch ] ---> [ CLIP Vision Encoder ] ---> [ 512-Dimensional Vector ] ---> [ Pgvector / Qdrant ]
^
| (Cosine Similarity)
v
[ "Sunset over temple" ] -> [ CLIP Text Encoder ] ---> [ 512-Dimensional Vector ] ------------+
Setting Up the Stack: Python, PyTorch, and Qdrant
For this project, we are going to use Python because of its rich ecosystem for machine learning, Hugging Face's transformers library for loading the CLIP model, and Qdrant as our vector database (though you could easily swap this for pgvector or Milvus).
First, let's install our dependencies. Open your terminal and run:
pip install sentence-transformers qdrant-client pillow requestsWe use sentence-transformers because it provides a highly optimized, developer-friendly wrapper around Hugging Face models, making it incredibly easy to generate multimodal embeddings in just a few lines of code.
Step 1: Initializing the Vector Database
Let's write our database initialization script. We'll spin up an in-memory instance of Qdrant for development purposes, but in production, you would point this to a Docker container or Qdrant Cloud.
We need to create a collection. Our model of choice, clip-ViT-B-32, outputs vectors with 512 dimensions. We will use Cosine Distance as our similarity metric, which is ideal for comparing CLIP embeddings.
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
# Initialize Qdrant Client (In-Memory for development)
client = QdrantClient(":memory:")
COLLECTION_NAME = "historic_sketches"
# Create a collection with 512 dimensions (matching clip-ViT-B-32)
client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config=VectorParams(size=512, distance=Distance.COSINE),
)
print(f"Collection '{COLLECTION_NAME}' created successfully!")
Step 2: Generating Embeddings for Historical Sketches
Now, let's write the ingestion pipeline. We'll load the CLIP model and write a function that takes an image path (or URL), generates the vector embedding, and uploads it to Qdrant alongside some useful metadata (like the artist, year, and source URL).
from PIL import Image
from sentence_transformers import SentenceTransformer
import uuid
# Load the pre-trained CLIP model
# This model handles both text and images seamlessly
model = SentenceTransformer('clip-ViT-B-32')
def ingest_image(image_path: str, artist: str, title: str, year: str):
# Load the image
try:
if image_path.startswith("http"):
import requests
img = Image.open(requests.get(image_path, stream=True).raw)
else:
img = Image.open(image_path)
except Exception as e:
print(f"Failed to load image {image_path}: {e}")
return
# Generate the vector embedding
# The model processes the image and outputs a 512-dimension float array
vector = model.encode(img).tolist()
# Create a unique ID for our database point
point_id = str(uuid.uuid4())
# Upsert the vector and payload metadata into Qdrant
client.upsert(
collection_name=COLLECTION_NAME,
points=[
{
"id": point_id,
"vector": vector,
"payload": {
"title": title,
"artist": artist,
"year": year,
"image_url": image_path
}
}
]
)
print(f"Successfully indexed: '{title}' by {artist}")
Let's populate our database with a few mock historical entries (including a nod to Charlotte Canning's sketches of India):
# Simulating our database ingestion
ingest_image(
image_path="https://upload.wikimedia.org/wikipedia/commons/b/b9/Charlotte_Canning_sketch_1858.jpg", # Mock URL
artist="Charlotte Canning",
title="Sunset over the Ganges at Varanasi",
year="1858"
)
ingest_image(
image_path="https://upload.wikimedia.org/wikipedia/commons/3/3a/Taj_Mahal_old_drawing.jpg", # Mock URL
artist="Unknown British Artist",
title="Detailed architectural sketch of the Taj Mahal facade",
year="1840"
)
ingest_image(
image_path="https://upload.wikimedia.org/wikipedia/commons/1/12/Crowded_market_delhi.jpg", # Mock URL
artist="William Simpson",
title="Bustling marketplace in Old Delhi",
year="1867"
)
Step 3: Building the Semantic Search Query
This is where the magic happens. A user comes to our web application and types: "A peaceful river scene at dusk."
Note that none of our metadata contains those exact words. However, because our CLIP model understands the semantic concepts of "peaceful," "river," and "dusk," it will map this text query directly to the vector space right next to Charlotte Canning's sketch of the Ganges at sunset.
Here is how we query our database using a natural language string:
def search_sketches(query_text: str, limit: int = 2):
# 1. Convert the search text into a vector embedding using the SAME model
query_vector = model.encode(query_text).tolist()
# 2. Query Qdrant for the closest vectors based on Cosine Similarity
search_results = client.search(
collection_name=COLLECTION_NAME,
query_vector=query_vector,
limit=limit
)
# 3. Print the results
print(f"\n--- Search Results for: '{query_text}' ---")
for result in search_results:
payload = result.payload
score = result.score
print(f"Match: '{payload['title']}' by {payload['artist']} ({payload['year']})")
print(f"Similarity Confidence Score: {score:.4f}")
print(f"Source: {payload['image_url']}\n")
# Let's run a test query!
search_sketches("drawings of riverside sunsets in India")
If you run this code, you will see that "Sunset over the Ganges at Varanasi" bubbles up to the very top with a high similarity score, despite the database having zero keywords matching "riverside" or "drawings" in the metadata fields. This is the power of vector embeddings!
Engineering Considerations: Handling Historical Artifacts in Production
While the implementation above is straightforward, deploying vector search for historical, non-photographic artwork in production brings unique challenges that you must design for:
1. Domain-Specific Model Fine-Tuning
Standard models like CLIP are trained predominantly on modern digital photographs. They are highly accurate with modern items but can struggle with the stylistic nuances of 19th-century sketches, copperplate engravings, and faded watercolor washes. If accuracy drops, you should look into fine-tuning your model using a contrastive loss framework on historical datasets like those provided by the British Library or the Rijksmuseum API.
2. Vector Index Optimization
As your archive grows from hundreds of sketches to millions of digitized assets, flat searches (exact comparisons) become too slow and CPU-intensive. You must configure **HNSW (Hierarchical Navigable Small World)** graphs in your vector database. HNSW creates a multi-layered graph structure that allows for approximate nearest neighbor (ANN) searches in logarithmic time $O(\log N)$, keeping search latency under 50ms even with millions of high-dimensional vectors.
3. Hybrid Search: Combining Vector and Metadata Filters
In real-world applications, users don't just want semantic search; they want to filter by date ranges, specific collections, or artists. You should implement **Hybrid Search**, which combines semantic vector search with boolean payload filtering. In Qdrant, this is done using a query filter that is evaluated during the vector graph traversal, preventing search-space pollution.
Conclusion: The Future of Historical Discovery
The digitization of physical history is moving at an incredible pace. Thanks to developments in machine learning and database technology, the work of historical artists like Charlotte Canning is no longer trapped behind manual, tedious cataloging. By utilizing vision-language models and vector databases, we can build tools that allow historians, students, and curious developers to browse human history through conceptual, emotional, and visual connections.
The code we wrote today is the exact architectural foundation used by major digital libraries and museums worldwide to bring ancient collections to life in the web browser.
What are your thoughts? Have you worked with vector databases or multimodal models in your projects? Are you using pgvector, Qdrant, or Pinecone? Let’s chat in the comments section below!
Until next time, keep coding, keep building, and keep learning.
— Alex