Hey everyone, Alex here. Welcome back to "Coding with Alex" at sysseder.com!
Every now and then, a quirky link pops up on Hacker News that stops me dead in my tracks. Today, it was a post titled "Every exterior shot in The Taking of Pelham 123"—a meticulous, frame-by-frame cataloging of the iconic 1974 NYC transit thriller. While film buffs are celebrating this as a triumph of cinema preservation, my developer brain immediately went somewhere else: How do we build this programmatically, at scale, without losing our minds?
If you’ve ever had to build a media asset manager, a security footage analyzer, or an e-commerce visual search tool, you know that traditional metadata tagging is a nightmare. Manual tagging doesn't scale, and classic keyword searches fail when someone asks, "Show me all shots of NYC subway entrances from a low angle."
In this post, we’re going to build a modern, AI-powered spatial and visual video search engine. We will use Python, OpenCV for frame extraction, a pre-trained CLIP model from OpenAI to generate multimodal vector embeddings (translating both images and text into the same vector space), and Qdrant as our vector database. By the end of this guide, you’ll know how to search through hours of video using natural language queries like "a vintage subway train station" or "a man in a brown trench coat."
The Architecture: From Raw Video to Semantic Search
To build a semantic video search engine, we can't just rely on OCR or basic color histograms. We need to understand the meaning of the frames. Here is how our pipeline will look:
- Frame Extraction (Keyframing): We ingest the raw video file. Instead of analyzing all 24 frames per second (which is computationally expensive and redundant), we use scene change detection or fixed-interval sampling to extract keyframes.
- Embedding Generation: We pass these keyframes through OpenAI's CLIP (Contrastive Language-Image Pre-training) model. CLIP is incredible because it maps images and text descriptions to the same vector space. An image of a yellow cab and the text "a yellow taxi cab" will yield vectors that are incredibly close to each other.
- Vector Indexing: We store these high-dimensional vectors (512 dimensions for CLIP-ViT-B-32) in Qdrant, an open-source vector database designed for fast similarity searches.
- Querying: When a user types a search query, we embed the text query using the same CLIP model and perform a cosine similarity search against our indexed video frames.
Here is a simple textual representation of our data flow:
[ Raw Video ] ──> (OpenCV Frame Extraction) ──> [ Keyframes ]
│
▼
[ User Search Query ] ──> (CLIP Text Encoder) [ CLIP Image Encoder ]
│ │
▼ ▼
[ Query Vector ] ───( Cosine Similarity )───> [ Vector Database (Qdrant) ]
│
▼
[ Best Matching Frames ]
Setting Up Your Environment
Let's spin up a clean Python environment and install our dependencies. We’ll need OpenCV for handling video, PyTorch and Hugging Face's Transformers for CLIP, and the Qdrant client.
pip install opencv-python pillow transformers torch qdrant-client tqdmWe'll also run a local instance of Qdrant using Docker. It’s lightweight, incredibly fast, and perfect for development:
docker run -p 6333:6333 qdrant/qdrantStep 1: Smart Keyframe Extraction with OpenCV
First, let’s write a utility to extract frames. While we could extract every single frame, movies have a lot of visual redundancy. For our Pelham 123 inspired search engine, extracting one frame every 1 to 2 seconds of video is more than enough to capture every exterior shot transition.
import cv2
import os
from PIL import Image
def extract_keyframes(video_path, output_dir, interval_seconds=1.0):
if not os.path.exists(output_dir):
os.makedirs(output_dir)
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
# Calculate how many frames to skip
frame_interval = int(fps * interval_seconds)
frame_count = 0
saved_count = 0
extracted_frames = []
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
if frame_count % frame_interval == 0:
# Convert BGR (OpenCV) to RGB (PIL)
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
pil_img = Image.fromarray(rgb_frame)
timestamp = frame_count / fps
frame_filename = f"frame_{saved_count:04d}_ts_{timestamp:.2f}.jpg"
frame_filepath = os.path.join(output_dir, frame_filename)
pil_img.save(frame_filepath)
extracted_frames.append({
"filepath": frame_filepath,
"timestamp": timestamp,
"image": pil_img
})
saved_count += 1
frame_count += 1
cap.release()
print(f"Extracted {saved_count} frames from {video_path}.")
return extracted_frames
Step 2: Vectorizing Frames with CLIP
Now that we have our frames, we need to convert them into mathematical representations that capture semantic meaning. We'll use Hugging Face's implementation of the clip-vit-base-patch32 model.
import torch
from transformers import CLIPProcessor, CLIPModel
# Initialize CLIP model and processor
device = "cuda" if torch.cuda.is_available() else "cpu"
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
def generate_image_embedding(pil_image):
inputs = processor(images=pil_image, return_tensors="pt").to(device)
with torch.no_grad():
image_features = model.get_image_features(**inputs)
# Normalize vector to unit length
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
return image_features.cpu().numpy()[0].tolist()
Step 3: Indexing Vectors in Qdrant
With our embeddings ready, we need to spin up our connection to Qdrant, define a collection (essentially our database table), and upsert our image vectors along with metadata (the timestamp and file path) so we can easily retrieve them later.
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
# Initialize connection to local Qdrant container
client = QdrantClient(url="http://localhost:6333")
COLLECTION_NAME = "pelham_shots"
# Configure collection (CLIP output dimension is 512)
client.recreate_collection(
collection_name=COLLECTION_NAME,
vectors_config=VectorParams(size=512, distance=Distance.COSINE),
)
def index_video_frames(frames):
points = []
for i, frame in enumerate(frames):
print(f"Embedding frame {i+1}/{len(frames)}...")
vector = generate_image_embedding(frame["image"])
# We store metadata (payload) along with the vector
payload = {
"filepath": frame["filepath"],
"timestamp": frame["timestamp"]
}
points.append(PointStruct(id=i, vector=vector, payload=payload))
# Bulk upsert to Qdrant
client.upsert(collection_name=COLLECTION_NAME, points=points)
print("Vector database indexing complete!")
Step 4: Implementing Multi-Modal Semantic Search
The magic of CLIP is that we can search our database of images using a text string because the model places semantically similar concepts together. Let’s write a function to embed our text search query and query Qdrant.
def generate_text_embedding(text_query):
inputs = processor(text=[text_query], return_tensors="pt", padding=True).to(device)
with torch.no_grad():
text_features = model.get_text_features(**inputs)
# Normalize vector to unit length
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
return text_features.cpu().numpy()[0].tolist()
def search_video(query, limit=3):
query_vector = generate_text_embedding(query)
# Query Qdrant for the closest matching vectors
search_results = client.search(
collection_name=COLLECTION_NAME,
query_vector=query_vector,
limit=limit
)
print(f"\n--- Search Results for: '{query}' ---")
for hit in search_results:
print(f"Score: {hit.score:.4f} | Frame Time: {hit.payload['timestamp']:.2f}s | Path: {hit.payload['filepath']}")
Putting It All Together
Let's run a complete scenario. Say we have a movie clip named pelham_123.mp4. We can process it, index it, and execute query searches in just a few lines of code.
if __name__ == "__main__":
# 1. Extract frames from our video file
video_file = "pelham_123.mp4" # Replace with your target video
output_directory = "./extracted_frames"
# Extract frame every 1 second of video
frames = extract_keyframes(video_file, output_directory, interval_seconds=1.0)
# 2. Index frames into Qdrant Vector DB
index_video_frames(frames)
# 3. Perform natural language queries
search_video("a retro subway platform with passengers")
search_video("a police officer wearing a hat")
search_video("an aerial view of a city street")
Why This Matters for Modern Developers
What makes this approach so powerful is its extreme flexibility. Think about how much logic you would have to write to parse video frames using traditional classification models (e.g., Object Detection with YOLO, OCR engines, Face Recognition). You would end up with a brittle pipeline of 4 different models stitched together.
With CLIP and Vector Databases, we bypass hand-crafted classification altogether. The model already "knows" what a vintage train, a wet street, or a dark shadow looks like from its massive pre-training. And because vector databases like Qdrant can scale to billions of vectors with sub-millisecond search times, you can scale this pipeline from a single movie to entire video archives.
Going Production-Ready
If you're looking to run this in a production cloud environment, here are a few optimizations to keep in mind:
- Batch Processing: Don't embed images one-by-one. Batch your frames together (e.g., in sizes of 16 or 32) when passing them to CLIP to maximize GPU utilization.
- Adaptive Keyframing: Instead of simple intervals, use pixel-difference histograms or deep-learning-based scene change detectors (like
PySceneDetect) to extract keyframes only when the shot actually changes. - Hybrid Search: Combine vector similarity with traditional metadata search (e.g., filtering search results by movie genre, camera type, or timestamp range) using Qdrant’s payload filtering features.
Wrapping Up
What started as a fun Hacker News rabbit hole cataloging 1970s NYC film shots highlights just how accessible advanced machine learning pipelines have become for everyday developers. You don’t need a PhD in Computer Vision to build a production-grade, semantic video search engine anymore—just a few lines of Python, the right vector embedding model, and a solid vector database.
Are you building anything with vector search or video indexing? What strategies are you using to handle large-scale media assets? Let me know in the comments below, or hit me up on Twitter!
Until next time, happy coding!