Hey everyone, Alex here. Welcome back to another post on Coding with Alex at sysseder.com!
If you have been keeping an eye on the tech landscape lately, you have probably noticed that "Data Science" and "Machine Learning" are no longer niche specialties reserved for academic researchers with PhDs. Today, software engineers are being asked to build, deploy, and maintain systems powered by AI. Whether you are implementing semantic search for an e-commerce platform, building a recommendation engine, or setting up a Retrieval-Augmented Generation (RAG) pipeline for a Large Language Model (LLM), you are dealing with data science.
But here is the secret: underneath the fancy marketing buzzwords of modern AI lies a bedrock of fundamental mathematics. When we talk about vector databases, embeddings, and semantic search, we are actually talking about linear algebra and geometry.
If you are like me, your eyes might glaze over when you see a wall of Greek mathematical symbols. But as developers, we have a superpower: we can translate abstract math into concrete code to truly understand how it works. Today, we are going to demystify the Mathematics of Data Science by exploring the core concept of Vector Spaces and implementing Cosine Similarity from scratch in Python. No black-box libraries, no magic—just pure, logical code.
Why Vector Math Rules the Modern Stack
Before we write any code, let's address the "why." Why do we represent data as vectors in a high-dimensional space?
Computers are famously terrible at understanding the meaning of things. They are great at comparing numbers (is 5 > 3?) and strings (does "apple" == "apple"?), but they struggle with conceptual relationships. If a user searches your database for "feline companion," a traditional SQL query using LIKE "%feline companion%" won't return a document containing "domestic cat," even though they mean the exact same thing.
This is where embeddings come in. An embedding model takes real-world data (text, images, audio) and translates it into an array of floating-point numbers—a vector. Crucially, the model is trained so that concepts that are semantically similar in the real world are placed close together in this mathematical "vector space."
Imagine a simplified 2D vector space where the X-axis represents "How feline is this?" and the Y-axis represents "How domestic is this?" on a scale of 0 to 1:
- Lion:
[0.9, 0.1](Highly feline, not domestic) - House Cat:
[0.95, 0.95](Highly feline, highly domestic) - Golden Retriever:
[0.1, 0.9](Not feline, highly domestic)
By converting these concepts into coordinates, we can use standard geometry to calculate the "distance" between them. The closer the vectors, the more similar the concepts.
The Math: Cosine Similarity vs. Euclidean Distance
How do we actually measure the "distance" between two vectors? There are two primary mathematical tools for this: Euclidean Distance and Cosine Similarity.
Euclidean Distance
Euclidean distance is the "straight-line" distance between two points in space. You probably remember this from high school geometry (the Pythagorean theorem). While useful, Euclidean distance has a major flaw in data science: it is highly sensitive to the magnitude (length) of the vectors.
For example, if you are comparing two text documents, a short document about cats and a massive textbook about cats will have very different vector lengths simply because the textbook repeats the word "cat" thousands of times. They point in the same direction conceptually, but Euclidean distance would register them as being far apart.
Cosine Similarity
This is where Cosine Similarity shines. Instead of measuring the distance between the endpoints of two vectors, it measures the angle between them. It completely ignores the magnitude of the vectors and focuses purely on their direction.
The mathematical formula for the cosine of the angle $\theta$ between two vectors $A$ and $B$ is:
A · B
Cosine Similarity = -------
||A|| ||B||
Let's break that down into developer terms:
- A · B (The Dot Product): We multiply corresponding elements of vector A and vector B, and sum the results.
- ||A|| and ||B|| (The Norms/Magnitudes): The length of each vector, calculated by squaring each element, summing them, and taking the square root.
The result of this calculation is always a value between -1 and 1:
- 1: The vectors point in the exact same direction (perfectly similar).
- 0: The vectors are orthogonal (perpendicular / completely unrelated).
- -1: The vectors point in opposite directions (exactly opposite).
Implementing Cosine Similarity from Scratch
Now that we understand the math, let's write some code. We won't use NumPy or PyTorch here. Writing this in plain Python will help cement how the math translates to code loops.
import math
def dot_product(vector_a, vector_b):
"""Calculate the dot product of two vectors."""
if len(vector_a) != len(vector_b):
raise ValueError("Vectors must be of the same dimension")
return sum(a * b for a, b in zip(vector_a, vector_b))
def magnitude(vector):
"""Calculate the Euclidean norm (magnitude) of a vector."""
return math.sqrt(sum(x ** 2 for x in vector))
def cosine_similarity(vector_a, vector_b):
"""Calculate the cosine similarity between two vectors."""
mag_a = magnitude(vector_a)
mag_b = magnitude(vector_b)
# Prevent division by zero
if mag_a == 0 or mag_b == 0:
return 0.0
return dot_product(vector_a, vector_b) / (mag_a * mag_b)
# --- QUICK TEST ---
# Vector A: "Cat" concept [0.9, 0.8]
# Vector B: "Kitten" concept [0.85, 0.85] (very similar)
# Vector C: "Space Shuttle" concept [0.1, 0.05] (very different)
vector_a = [0.9, 0.8]
vector_b = [0.85, 0.85]
vector_c = [0.1, 0.05]
print(f"Similarity (Cat, Kitten): {cosine_similarity(vector_a, vector_b):.4f}")
print(f"Similarity (Cat, Space Shuttle): {cosine_similarity(vector_a, vector_c):.4f}")
If you run this script, you will see output like this:
Similarity (Cat, Kitten): 0.9988
Similarity (Cat, Space Shuttle): 0.9234
Wait, why is the Space Shuttle similarity so high (0.92) when they are different? That is because in a 2D space where all values are positive, the angle between any two vectors is constrained to a 90-degree quadrant, limiting how different they can look. In a real-world machine learning application, embeddings have hundreds or thousands of dimensions (e.g., OpenAI's text-embedding-3-small has 1536 dimensions!), allowing for highly precise, multi-directional spatial segregation.
Building a Miniature Semantic Search Engine
Let's take this a step further. Let's build a mini-semantic search engine. We will define a small corpus of documents, mock their vector embeddings (simulating what an embedding API would return), and query them using our mathematical similarity function.
# A mock database of articles with 4-dimensional embeddings.
# Dimensions represent: [Tech, Cooking, Sports, Space]
documents = [
{
"title": "How to deploy a Docker container to AWS",
"embedding": [0.95, 0.05, 0.01, 0.1] # High Tech
},
{
"title": "The ultimate sourdough bread recipe",
"embedding": [0.02, 0.98, 0.05, 0.01] # High Cooking
},
{
"title": "Understanding the James Webb Space Telescope",
"embedding": [0.4, 0.01, 0.05, 0.95] # Medium Tech, High Space
},
{
"title": "World Cup qualifiers: tactics and analysis",
"embedding": [0.05, 0.1, 0.95, 0.02] # High Sports
}
]
def search(query_vector, corpus, top_n=2):
results = []
for doc in corpus:
score = cosine_similarity(query_vector, doc["embedding"])
results.append((doc["title"], score))
# Sort results by similarity score in descending order
results.sort(key=lambda x: x[1], reverse=True)
return results[:top_n]
# Scenario: User searches for "Kubernetes hosting and cloud orchestration"
# Our embedding model generates a tech-heavy query vector:
query_tech = [0.88, 0.02, 0.02, 0.2]
print("--- Search Results for 'Cloud Tech' ---")
for title, score in search(query_tech, documents):
print(f"[{score:.4f}] {title}")
# Scenario: User searches for "NASA Mars mission updates"
# Our embedding model generates a space-heavy query vector:
query_space = [0.2, 0.0, 0.0, 0.9]
print("\n--- Search Results for 'Mars Mission' ---")
for title, score in search(query_space, documents):
print(f"[{score:.4f}] {title}")
When you run this search engine, the output perfectly aligns with semantic intent:
--- Search Results for 'Cloud Tech' ---
[0.9995] How to deploy a Docker container to AWS
[0.5511] Understanding the James Webb Space Telescope
--- Search Results for 'Mars Mission' ---
[0.9890] Understanding the James Webb Space Telescope
[0.3455] How to deploy a Docker container to AWS
Without using any keyword matching (the word "Docker" or "AWS" was not in our cloud query, nor was "telescope" in our space query), our simple mathematical implementation correctly mapped the concepts together!
Scale Challenges: Enter Vector Databases
You might be thinking: "Alex, this is incredibly simple. Why do we need heavy infrastructure like Pinecone, Milvus, Qdrant, or pgvector?"
The answer is computational complexity. In our search function above, we had to compare our query vector against every single document in our database. This is known as a linear search, or an $O(N)$ operations complexity.
If you have 100 documents, it takes milliseconds. If you have 10,000,000 documents, and each embedding has 1,536 dimensions, your CPU will melt. Performing 10 million dot products on every single user query is simply not viable for real-time web applications.
Vector databases solve this using Approximate Nearest Neighbor (ANN) search algorithms. Instead of comparing the query to every single vector, they partition the vector space using graphs, trees, or quantization (like Hierarchical Navigable Small World, or HNSW). They trade a tiny bit of mathematical precision for massive boosts in speed, lowering the complexity from $O(N)$ to $O(\log N)$.
Conclusion
Data science and modern AI can often feel like wizardry shrouded in complex academic terminology. But beneath it all, it is just developers manipulating arrays, calculating angles, and optimizing pipelines. By understanding how vector embeddings and cosine similarity function under the hood, you are no longer just gluing APIs together—you are actively designing how your systems process semantic meaning.
The next time you spin up a vector database or write a prompt for an LLM agent, you'll know exactly what's happening when those embeddings are compared in the background.
What about you? Are you working on integrating AI, semantic search, or RAG systems into your stack? Have you run into scaling issues with vector queries? Let’s talk about it in the comments below!
Until next time, happy coding!