Hey everyone, Alex here. Welcome back to another edition of Coding with Alex on sysseder.com.
If you’ve been hanging out in developer circles lately, you probably noticed a classic academic textbook suddenly climbing its way to the top of the Hacker News homepage: "Linear Algebra Done Right" by Sheldon Axler. This isn’t a brand-new release—it’s a legendary math text. But its sudden resurgence in the developer community highlights a massive shift happening in our industry right now.
We are officially moving past the "wrapper era" of AI. A year or two ago, you could build a successful startup or a killer feature just by making API calls to OpenAI or Anthropic. Today, those low-hanging fruits are gone. If you want to build truly competitive software in 2024 and beyond, you have to write custom embedding pipelines, optimize vector database queries, fine-tune open-source LLMs like Llama 3, and write custom CUDA kernels.
And guess what powers all of that? Linear algebra.
But here is the catch: most of us were taught linear algebra wrong. We were taught to memorize rote algorithms for computing determinants of 3x3 matrices by hand, or performing endless row reductions. That approach is boring, tedious, and completely misses the point for software engineers. Sheldon Axler’s book is famous because it takes a radically different approach: it avoids determinants entirely and focuses on linear operators, vector spaces, and geometric intuition.
Today, we’re going to look at why Axler's approach to linear algebra is exactly what you need to upgrade your mind from a "framework consumer" to a "systems engineer" who can build modern AI and graphics applications from scratch. We'll also translate some of these abstract mathematical concepts into practical Python and NumPy code so you can see exactly how they apply to your day-to-day work.
The Determinant-Free Philosophy: Why Developers Love Axler
If you took linear algebra in college, your professor probably introduced the determinant in the first week. You were given a formula, told to cross-multiply terms, and spent hours calculating values that supposedly told you if a matrix was invertible.
Axler argues that this is "done wrong." In his preface, he writes that determinants are "difficult, non-intuitive, and defined without mnemonic value." Instead, he focuses on Linear Operators—transformations that map vectors from one space to another while preserving vector addition and scalar multiplication.
As developers, this should instantly resonate with you. A linear operator is essentially a pure function. It takes an input (a vector), performs a transformation, and returns an output (another vector), with no side effects.
When we view linear algebra through the lens of operators rather than matrix grids, things like coordinate transformations, dimensionality reduction (PCA), and neural network layers suddenly make intuitive sense. We stop thinking of matrices as "boxes of numbers" and start viewing them as "functions that stretch, rotate, and project data."
From Math to Code: Representing Vector Spaces and Operators
Let's ground this in something we write every day: code. In Axler's world, everything starts with a vector space. A vector space is a collection of objects (vectors) that can be added together and multiplied by scalars (numbers), satisfying certain algebraic properties.
In Python, we often represent vectors as simple lists or NumPy arrays. But let's build a mental bridge by defining a basic linear operator. Suppose we want to define an operator $T$ that projects a 3D vector onto the 2D $xy$-plane. Mathematically, this is written as:
T(x, y, z) = (x, y, 0)In code, we can represent this operator both as a function and as a matrix multiplication. Let's see how NumPy handles this, and why the "operator" mental model is so clean:
import numpy as np
# Define our input vector in 3D space
vector_v = np.array([3.0, 4.0, 5.0])
# 1. The "Function/Operator" approach
def project_to_xy(vector):
return np.array([vector[0], vector[1], 0.0])
result_functional = project_to_xy(vector_v)
print(f"Functional Projection: {result_functional}") # Outputs: [3. 4. 0.]
# 2. The "Matrix/Linear Algebra" approach
# This matrix represents the operator T relative to the standard basis
projection_matrix = np.array([
[1.0, 0.0, 0.0],
[0.0, 1.0, 0.0],
[0.0, 0.0, 0.0]
])
result_matrix = np.dot(projection_matrix, vector_v)
print(f"Matrix Projection: {result_matrix}") # Outputs: [3. 4. 0.]
Notice how the matrix multiplication does the exact same work as our pure Python function. A matrix is simply a concrete, computational representation of a abstract linear operator. When you run a forward pass in a PyTorch model, you are running millions of these operator evaluations in parallel.
Why Vector Spaces Matter: The Math of LLM Embeddings
If you're building search engines, RAG (Retrieval-Augmented Generation) systems, or recommendation engines, you are working with embeddings. An embedding is just a vector in a high-dimensional space that represents the "semantic meaning" of a piece of text, an image, or a user's behavior.
When you use OpenAI's text-embedding-3-small model, it outputs vectors with 1536 dimensions. That means your data lives in a 1536-dimensional vector space ($\mathbb{R}^{1536}$).
To find similar documents, we compute the cosine similarity between vectors. Let's look at how the dot product (a core concept in Axler's inner product spaces) is used to calculate this similarity. If you understand the geometry of inner product spaces, you understand exactly how search works under the hood:
import numpy as np
def cosine_similarity(v1, v2):
# The inner product (dot product) of v1 and v2
dot_product = np.dot(v1, v2)
# The norm (magnitude) of the vectors
norm_v1 = np.linalg.norm(v1)
norm_v2 = np.linalg.norm(v2)
# Cosine of the angle between them
return dot_product / (norm_v1 * norm_v2)
# Semantic vectors (hypothetical embeddings)
query_vector = np.array([0.12, 0.85, -0.34, 0.05])
doc_apple_fruit = np.array([0.15, 0.82, -0.30, 0.01])
doc_apple_stock = np.array([-0.65, 0.12, 0.78, 0.45])
sim_fruit = cosine_similarity(query_vector, doc_apple_fruit)
sim_stock = cosine_similarity(query_vector, doc_apple_stock)
print(f"Similarity to fruit document: {sim_fruit:.4f}") # High similarity (~0.99)
print(f"Similarity to financial document: {sim_stock:.4f}") # Low similarity
By understanding vector spaces, you realize that "semantic search" is just finding vectors that point in roughly the same direction within a multi-dimensional inner product space. If you want to optimize your vector database indexes (like HNSW or IVFFlat), you aren't just tweaking configuration files—you are tuning algorithms that navigate these mathematical spaces.
Eigenvalues and Eigenvectors: The Heart of PageRank and LLM Attention
One of the absolute best parts of "Linear Algebra Done Right" is how it handles eigenvalues and eigenvectors. In standard textbooks, you calculate these by solving the characteristic equation $\det(A - \lambda I) = 0$. Axler bypasses this entirely, defining eigenvalues through the structure of invariant subspaces.
An eigenvector of a linear operator $T$ is a non-zero vector $v$ such that applying $T$ to $v$ only scales it. That is:
T(v) = λvWhere $\lambda$ (lambda) is the eigenvalue. In plain English: an eigenvector is a direction that does not change when the transformation is applied; the vector is only stretched or shrunk.
Why do developers care about this?
- Google's PageRank: The importance of web pages can be modeled as a transition matrix of probability. The PageRank of the entire web is simply the eigenvector corresponding to the eigenvalue of $\lambda = 1$ of this matrix.
- Principal Component Analysis (PCA): When we want to compress 1536-dimensional embeddings down to 2 or 3 dimensions for visualization, we find the eigenvectors of the data's covariance matrix. These point in the directions of maximum variance.
- Spectral Clustering: Used in image segmentation and graph analysis to identify clusters of data points based on eigenvalues of similarity graphs.
Let's use NumPy to extract the principal components (eigenvectors) of a toy dataset to see how PCA works mathematically:
import numpy as np
# Simulating some 2D data (e.g., house size vs. price)
np.random.seed(42)
x = np.random.normal(0, 1, 100)
y = 2 * x + np.random.normal(0, 0.5, 100) # Strong linear correlation
data = np.vstack((x, y)).T
# Center the data (mean = 0)
data_centered = data - np.mean(data, axis=0)
# Calculate the covariance matrix
covariance_matrix = np.cov(data_centered.T)
# Calculate eigenvalues and eigenvectors (Axler style!)
eigenvalues, eigenvectors = np.linalg.eig(covariance_matrix)
print("Eigenvalues:")
print(eigenvalues)
print("\nEigenvectors (Principal Directions):")
print(eigenvectors)
When you run this, the eigenvector with the largest eigenvalue points precisely in the direction along which the data is most stretched out. That is your "Principal Component 1." This is how we reduce features in machine learning pipelines without losing critical information.
Why Reading Axler Will Make You a Better Systems Engineer
You might be asking, "Alex, why should I read a math textbook when PyTorch, JAX, and NumPy do all this math for me?"
It's about debugging and optimization. When you write high-performance code, you can't treat libraries as magic black boxes.
For instance, if you are writing a custom attention mechanism in PyTorch for a transformer model, you are multiplying Query, Key, and Value matrices ($QK^T / \sqrt{d_k} V$). If your model is running out of memory (VRAM), understanding the dimension transformations, matrix ranks, and memory layouts (row-major vs. column-major) allows you to use techniques like FlashAttention—which is fundamentally a clever mathematical rewrite of the attention algorithm to optimize hardware cache access.
Furthermore, graphics programming (WebGPU, Vulkan, Metal) relies entirely on 4x4 transform matrices. If you don't understand how operators compose, you will spend days debugging weirdly rotated 3D meshes or broken camera projections.
Conclusion: Time to Open the Textbook
The tech landscape is shifting. The developers who thrive in the next decade won't just be those who can glue APIs together; they will be the ones who understand the fundamental mathematical and physical principles powering our tools.
Sheldon Axler's "Linear Algebra Done Right" (now in its 4th edition, which is beautifully formatted and free in PDF format from Springer!) is the perfect portal to build this deeper intuition. It is rigorous, elegant, and completely alters the way you think about multidimensional space.
So, here is my challenge to you this week: step away from the Javascript frameworks for a weekend. Download a copy of Axler's book, grab a notebook, and read the first chapter on Vector Spaces. You'll be amazed at how quickly those mathematical abstractions begin to illuminate your code.
Have you read "Linear Algebra Done Right"? How has a strong math background helped you in your software engineering career? Let's discuss in the comments below!